Add build caching to docker #810

asutula · 2021-03-21T05:31:08Z

Still trying to understand how exactly this works and what exactly BuildKit is, but this simple change seems too good to be true. Caches build objects and speeds up docker builds by many many times. Based on what I was reading in this blog post and the others in the same series: https://www.docker.com/blog/containerize-your-go-developer-environment-part-2/

Signed-off-by: Aaron Sutula <hi@asutula.com>

eightysteele · 2021-03-21T06:57:32Z

yessss, so good. def not an expert here, but the original build backend to docker of course had this annoying sequential-build-step-execution thing. fortunately the "multi-stage build" stuff made things a little better by grouping build stages into logical "build tasks".... and those "build tasks" had the nice property of being totally independent, which means in theory they could be executed in parallel. so i think that's what buildkit takes advantage of. it creates a graph of those dependencies between "build tasks" so that it can easily track which parts of the build can be totally ignored, executed in parallel vs. sequentially, etc. assuming that's correct, it's a pretty nice optimization. yay graphs. :)

jsign

Nice!

The explanation is actually much simpler :)

The Go compiler has a build-cache that it's saved in GOCACHE environment variable path folder. It does that's done by default and not related to Docker.

What this change is doing, is caching that folder in a docker cache, similar-ish to the onion-layer caching that Docker does but you can select the target folder (in this case, the $GOCACHE folder).

This is basically what happens if you compile things in your own machine. You have previously built dependencies cached in $GOCACHE, so next time it builds the binary, it leverages work existing there.

jsign · 2021-03-21T12:51:02Z

yessss, so good. def not an expert here, but the original build backend to docker of course had this annoying sequential-build-step-execution thing. fortunately the "multi-stage build" stuff made things a little better by grouping build stages into logical "build tasks".... and those "build tasks" had the nice property of being totally independent, which means in theory they could be executed in parallel. so i think that's what buildkit takes advantage of. it creates a graph of those dependencies between "build tasks" so that it can easily track which parts of the build can be totally ignored, executed in parallel vs. sequentially, etc. assuming that's correct, it's a pretty nice optimization. yay graphs. :)

Yep, that's correct but there's no much room for parallelization in our build pipelines (in all our repos). :)
The slowest part is downloading dependencies, and just go build, and those parts need to be done serially anyway.

What we really leverage of multi-stage pipelines is building the final artifact in a smaller base-image without any temporal/garbage side-products (dependencies, build caches, source code) of the build stage. So the final Docker image is simply a small base image, plus only the binary from the build stage.

What this PR does to make things faster, is having a way to cache part of the RUN go build step which always gets invalidated when the source-code changes. That's because the build cache is in the same Docker layer as the generated binary, so without these new flags, anything that invalidates the cached binary (i.e: source code change) would also invalidate the go-cache (which is unfortunate, but with these new flags we're kind of untangling that).

eightysteele · 2021-03-21T16:28:33Z

What this PR does to make things faster, is having a way to cache part of the RUN go build step which always gets invalidated when the source-code changes

Oh interesting! I'm definitely learning something new about builds here. :)

@asutula super curious, do you a have a sense for how much faster it is after the change?

sanderpick · 2021-03-21T16:36:46Z

Maybe related to my version of Docker Desktop, but I'm getting caching out-of-the-box:

> env REPO_PATH=~/.buckets make buck-up
docker-compose -f cmd/buckd/docker-compose-dev.yml up --build
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Building buckets
[+] Building 3.0s (18/18) FINISHED
 => [internal] load build definition from Dockerfile.dev                                                                                                                         0.5s
 => => transferring dockerfile: 41B                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                0.3s
 => => transferring context: 2B                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/debian:buster                                                                                                                 1.6s
 => [internal] load metadata for docker.io/library/golang:1.16.0-buster                                                                                                          0.0s
 => [auth] library/debian:pull token for registry-1.docker.io                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                0.4s
 => => transferring context: 117.13kB                                                                                                                                            0.1s
 => [stage-1 1/4] FROM docker.io/library/debian:buster@sha256:9d4ab94af82b2567c272c7f47fa1204cd9b40914704213f1c257c44042f82aac                                                   0.0s
 => [stage-0 1/7] FROM docker.io/library/golang:1.16.0-buster                                                                                                                    0.0s
 => CACHED [stage-0 2/7] RUN apt-get update                                                                                                                                      0.0s
 => CACHED [stage-0 3/7] RUN go get github.com/go-delve/delve/cmd/dlv                                                                                                            0.0s
 => CACHED [stage-0 4/7] COPY go.mod go.sum /go-buckets/                                                                                                                         0.0s
 => CACHED [stage-0 5/7] RUN cd /go-buckets   && go mod download                                                                                                                 0.0s
 => CACHED [stage-0 6/7] COPY . /go-buckets                                                                                                                                      0.0s
 => CACHED [stage-0 7/7] RUN cd /go-buckets   && CGO_ENABLED=0 GOOS=linux go build -gcflags "all=-N -l" -o buckd cmd/buckd/main.go                                               0.0s
 => CACHED [stage-1 2/4] COPY --from=0 /go/bin/dlv /usr/local/bin/dlv                                                                                                            0.0s
 => CACHED [stage-1 3/4] COPY --from=0 /go-buckets/buckd /usr/local/bin/buckd                                                                                                    0.0s
 => CACHED [stage-1 4/4] RUN adduser --home /data/buckets --disabled-login --gecos "" --ingroup users buckets                                                                    0.0s
 => exporting to image                                                                                                                                                           0.2s
 => => exporting layers                                                                                                                                                          0.0s
 => => writing image sha256:6ae872c31a3cd27c2a39ad01eb5786e25997b45fd0b67559c58fdcdebebda65e                                                                                     0.0s
 => => naming to docker.io/library/buckd_buckets                                                                                                                                 0.0s
Successfully built 6ae872c31a3cd27c2a39ad01eb5786e25997b45fd0b67559c58fdcdebebda65e
Starting buckd_ipfs_1    ... done
Starting buckd_threads_1 ... done
Starting buckd_buckets_1 ... done
Attaching to buckd_threads_1, buckd_ipfs_1, buckd_buckets_1

See following log output gives a hint:

Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/

Following the link in the message...

Compose by default uses the docker CLI to perform builds (also known as “native build”). By using the docker CLI, Compose can take advantage of features such as BuildKit, which are not supported by Compose itself. BuildKit is enabled by default on Docker Desktop, but requires the DOCKER_BUILDKIT=1 environment variable to be set on other platforms.

BuildKit is enabled by default. Output of docker version:

Client: Docker Engine - Community
 Cloud integration: 1.0.8
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        48d30b5
 Built:             Fri Jan 29 14:28:27 2021
 OS/Arch:           darwin/amd64 (rosetta)
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       46229ca
  Built:            Fri Jan 29 14:32:07 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

This is the Apple M1 alpha: https://docs.docker.com/docker-for-mac/apple-m1/

asutula · 2021-03-22T17:33:02Z

@eightysteele re: your question

do you a have a sense for how much faster it is after the change?

For building Powergate on my machine, it was taking about 1 min with no caching. With a cache in place and no code changes, the build was instantaneous. If I changed code in one file, the build takes about 2 seconds.

asutula · 2021-03-23T00:27:09Z

@sanderpick how can you tell from that output that BuildKit is enabled by default for you?

sanderpick · 2021-03-23T00:35:23Z

The output says "Building with native build." and provides a link that describes what native build means: Building with the docker CLI, which it says has BuildKit enabled by default.

asutula · 2021-03-23T00:44:20Z

Ah gotcha, thanks.

Signed-off-by: Aaron Sutula <hi@asutula.com>

…ergate into asutula/docker-build-cache

add build caching to docker

bd97e9f

Signed-off-by: Aaron Sutula <hi@asutula.com>

asutula self-assigned this Mar 21, 2021

jsign approved these changes Mar 21, 2021

View reviewed changes

Merge branch 'master' into asutula/docker-build-cache

dca7e11

asutula added 2 commits March 23, 2021 17:16

adding DOCKER_BUILDKIT=1 to Makefile

a914544

Signed-off-by: Aaron Sutula <hi@asutula.com>

Merge branch 'asutula/docker-build-cache' of github.com:textileio/pow…

76d973a

…ergate into asutula/docker-build-cache

asutula merged commit 4778f52 into master Mar 24, 2021

asutula deleted the asutula/docker-build-cache branch March 24, 2021 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add build caching to docker #810

Add build caching to docker #810

Uh oh!

asutula commented Mar 21, 2021

Uh oh!

eightysteele commented Mar 21, 2021 •

edited

Loading

Uh oh!

jsign left a comment

Uh oh!

jsign commented Mar 21, 2021 •

edited

Loading

Uh oh!

eightysteele commented Mar 21, 2021

Uh oh!

sanderpick commented Mar 21, 2021 •

edited

Loading

Uh oh!

asutula commented Mar 22, 2021 •

edited

Loading

Uh oh!

asutula commented Mar 23, 2021

Uh oh!

sanderpick commented Mar 23, 2021 •

edited

Loading

Uh oh!

asutula commented Mar 23, 2021

Uh oh!

Uh oh!

Add build caching to docker #810

Add build caching to docker #810

Uh oh!

Conversation

asutula commented Mar 21, 2021

Uh oh!

eightysteele commented Mar 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsign left a comment

Choose a reason for hiding this comment

Uh oh!

jsign commented Mar 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eightysteele commented Mar 21, 2021

Uh oh!

sanderpick commented Mar 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asutula commented Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asutula commented Mar 23, 2021

Uh oh!

sanderpick commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asutula commented Mar 23, 2021

Uh oh!

Uh oh!

eightysteele commented Mar 21, 2021 •

edited

Loading

jsign commented Mar 21, 2021 •

edited

Loading

sanderpick commented Mar 21, 2021 •

edited

Loading

asutula commented Mar 22, 2021 •

edited

Loading

sanderpick commented Mar 23, 2021 •

edited

Loading