Roadmap (keeping this up to date):

I think the natural way forward for us is:

• Move native Windows tasks to GitHub Actions (ci: Run "Windows (VS 2022)" job on GitHub Actions #1389 and ci: Remove "Windows (VS 2022)" task from Cirrus CI #1397)
• Move SageMath task to GitHub Actions (ci, gha: Run "SageMath prover" job on GitHub Actions #1399)
• Move native macOS tasks to GitHub Actions, this will convert them to x86_64 unfortunately (ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions #1394, ci: Remove "arm64: macOS Ventura" task from Cirrus CI #1404)
• Move Linux tasks to the Bitcoin Core persistent workers or alternatively to GitHub Actions
  • wine/msvc tasks (converting them to native windows in ci, gha: Run all MSVC tests on Windows natively #1401)
  • wine/mingw tasks (ci, gha: Add Windows jobs based on Linux image #1398)
  • actual normal Linux tasks (ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job #1396)
  • special tasks like qemu/sanitizers (ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions #1406, ci: Move remained task from Cirrus to GitHub Actions #1409)

Possible follow-ups:

• Consider using artifacts to move the Docker image from the Docker build job to the actual CI job (for Linux tasks). (Not worth the hassle, the current approach seems to work well.)
  Details

  
  This should be a cleaner solution, but it adds some complexity. It's also worth checking if this avoids network issues. In terms of delay, this adds about 12 min uploading time to the Docker build job, but avoids about 1 min delay in the actual CI jobs as compared to the current solution that relies purely on the GHA cache (ci, gha: Add Windows jobs based on Linux image #1398). So this will speed up CI if we could avoid re-uploading existing artifacts, e.g., have another digest file that just stores the SHA256, and re-upload only if the SHA does not match. But all of this is probably not worth the complexity if the current approach with the cache turns out to be good enough.
• Enable Valgrind on macOS again now that the macOS tasks run on x86_64 (Constant-time tests on macOS #1151, ci: Switch macOS from Ventura to Monterey and add Valgrind #1412)
• Bring back some ARM testing (see for details: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions #1394 (comment))
  • Add a task for ctimetest on ARM64/Linux/Valgrind on Cirrus CI using free minutes or the self-hosted runner (ci/cirrus: Add native ARM64 jobs #1426)
  • Add more ARM64 tests on QEMU (ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot #1414)
  • Add ARM32 tests (ci/cirrus: Add native ARM64 jobs #1426 (comment))
• Consider moving the git safe directory stuff to run-in-docker-action (ci: Make repetitive command the default one #1411)
• After the migration, check if the build matrix still makes sense
• cosmetics: Job names could need a rework
• cosmetics: Printing of log files could be improved

Other related PRs:

• ci, gha: Ensure only a single workflow processes github.ref at a time #1403
• ci: Use concurrency for pull requests only #1410

Corresponding Bitcoin Core issue: bitcoin/bitcoin#28098

Cirrus CI will cap the community cluster, see cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci. As with Core, the pricing model makes it totally unreasonable to pay for compute credits (multiple thousand USD / month).

The plan in Bitcoin Core is to move native Windows+macOS tasks to GitHub Actions, and move Linux tasks to persistent workers (=self-hosted). If I read the Bitcoin Core IRC meeting notes correctly, @MarcoFalke said these workers will also be available for libsecp256k1.

But the devil is in the details:

For macOS, we need to take also #1153 into account. It seems that GitHub-hosted macOS runners are on x86_64. The good news is that Valgrind should work again then, but the (very) bad is that this will reduce our number of native ARM tasks to zero. We still have some QEMU tasks, but we can't even the run the Valgrind cttimetests on them (maybe this would now work with MSan?!) @MarcoFalke Are the self-hosted runners only x86_64?

For Linux tasks, the meeting notes say that the main reason for using persistent workers is that some tasks require a very specific environment (e.g., the USDT ASan job). I don't think we have such requirements, so I tend to think that moving everything to GitHub Actions is a bit cleaner for us. With a persistent worker, Cirrus CI anyway acts only as a "coordination layer" between the worker and GitHub. Yet another way is to the self-hosted runners with GitHub Actions, see my comment bitcoin/bitcoin#28098 (comment)).