ci: allow passing extra args to docker build with CI_IMAGE_BUILD_EXTRA_ARGS #31377
Conversation
By setting `CI_IMAGE_BUILD_EXTRA_ARGS`, a CI runner can influence the docker build of the CI container image. This allows to, for example, pass `--cache-to` and `--cache-from` to the build, which is useful for ephemeral CI runners.
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31377. ReviewsSee the guideline for information on the review process. |
|
I've been looking into setting up my own ephemeral CI runners. Ideally, these would cache the CI container images that we build. By setting The current CI runners don't need this, as they aren't ephemeral. They cache the images in their internal docker image store which is reused in the next CI task. A general alternative would be to regularly build CI docker images, push them to a registry, and (cache-from)/pull them during the build. However, this seems to be a lot more effort. |
Currently, each runner is their own user account, with their own podman image layer cache, so in theory, they'd benefit from this as well. However, the images are quite large (especially msan), so my worry would be that after a sufficient number of builds, the workers will run out of storage, as the cache isn't cleared like the local images are. |
|
It occurred to me that the local cache needs to be keyed by e.g.
Yes, the local build cache dir seems to grow indefinitely as old layers aren't cleaned up. Based on the "cache versioning" documentation, I think we wouldn't need to keep old layers around as we wouldn't use the |
Are you sure? The final layers should be different for each task, so should never collide. (Possibly early layers may be shared, if they happen to be the same). If this was an issue, shouldn't the CI normally fail when running several tasks on the same machine in the same user account?
That seems racy, as one worker may still be using the cache when it is (re)moved, which could lead to errors? |
I'm not sure if we are talking about the same thing. I mean the My understanding is that {
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.index.v1+json",
"digest": "sha256:3a897fa438c7965a7a4919f2d55e5ea578221b4686ebc38f4fb843fac19fc3ec",
"size": 1559,
"annotations": {
"org.opencontainers.image.ref.name": "latest"
}
}
]
}With that, it will start importing the cache from
At least on Linux/Unix, when you open a file you get a file descriptor and can read from that descriptor even if the file is removed or overwritten with e.g. |
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
|
closing in favor of #31545 |
By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As --cache-to doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitly with old, unused layers. When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the cache type=local can be found https://docs.docker.com/build/cache/backends/local/ This replaces bitcoin#31377
e87429a ci: optionally use local docker build cache (0xb10c) Pull request description: By setting `DANGER_DOCKER_BUILD_CACHE_HOST_DIR`, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`. As `--cache-to` doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitely with old, unused layers. When `--cache-from` doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image. This feature is opt-in. The documentation for the docker build cache of `type=local` can be found on https://docs.docker.com/build/cache/backends/local/ This replaces #31377 - some of the discussion there might provide more context. ACKs for top commit: maflcko: I haven't tested this, and it looks harmless and is easy to revert, if needed. So lgtm ACK e87429a achow101: ACK e87429a TheCharlatan: tACK e87429a willcl-ark: ACK e87429a Tree-SHA512: 0887c395dee2e2020394933246d4c1bfb6dde7165219cbe93eccfe01379e05c75dce8920b6edd7df07364c703fcee7be4fba8fa45fd0e0e89da9e24759f67a71
By setting
CI_IMAGE_BUILD_EXTRA_ARGS, a CI runner can influence the docker build of the CI container image. This allows to, for example, pass--cache-toand--cache-fromto the build, which is useful for ephemeral, self-hosted CI runners.