I think we should work on implementing caching or appropriately scaling the runners instead. Is this mostly single threaded load? Why does it take so long? Could we get better performance with e.g. native ARM runners?

No idea about using other runners but I reduced the timeout from 12h I originally proposed to 8h and implemented some caching. It takes 6h+ with no/outdated cache and 4-5h with an up to date cache. The build time also vary due to the need to build some Guix modules when they are updated but 8h should be enough to cover that I think. Or we can slightly bump it later if needed.

8h seems extremely long to me, given we just got the (theoretically much more complex) platform build down to <10m. Can we cache the Guix modules that need to be built? Is there any way we can switch this back to running on GitHub-hosted runners?

8h seems extremely long to me, given we just got the (theoretically much more complex) platform build down to <10m.

8h is the upper limit, ~6.5h is the current known worst case scenario and this PR brings the time for (hopefully) most cases down to 4.5h. Also, in this Guix workflow we produce deterministic binaries for 7 hosts/platforms. I'm not familiar with platform workflows but is it even an apples-to-apples comparison?

Can we cache the Guix modules that need to be built?

We use substitutions (pre-built modules) already and I think all the cache Guix produces for it's own modules sits in .cache/guix which is cached too now.

Is there any way we can switch this back to running on GitHub-hosted runners?

I guess it should be possible to do it by using https://github.com/marketplace/actions/install-gnu-guix maybe (but probably won't help because we'd need to install other packages too somehow). Right now we use docker to build it (we do not install guix on host machines) and this PR simply aims to fix (and slightly optimise) Guix builds that were working previously. We could try to migrate to gh runners later imo.