7. BuildKit, caching and multi-stage builds

Chapter 6 traced a build. This chapter is about making builds fast, small and safe — three goals that turn out to have three specific mechanisms.

BuildKit has been the default builder since Docker 23. If you are on something older, set DOCKER_BUILDKIT=1, and check that nobody has set it to 0 in CI.

Multi-stage builds

The idea in one diagram:

   SINGLE STAGE                       MULTI-STAGE

   FROM golang:1.22                   FROM golang:1.22 AS build
   WORKDIR /src                       WORKDIR /src
   COPY . .                           COPY . .
   RUN go build -o /app ./cmd         RUN go build -o /app ./cmd
   CMD ["/app"]
                                      FROM scratch
                                      COPY --from=build /app /app
                                      CMD ["/app"]

   final image:  842 MB               final image:  11 MB
   contains: the Go toolchain,        contains: one binary
   the module cache, your source,
   git, and a compiler

Chapter 6’s step 5 is the reason: only the final stage’s layers become the image. Everything in earlier stages is a cache artifact.

The pattern generalizes to every compiled language and most interpreted ones:

   Go / Rust / C     build stage → scratch or distroless. 90%+ reduction.
   Java             build with Maven/Gradle → JRE-only or jlink runtime.
   Node             build (devDeps, TypeScript, bundler) → prod deps only.
   Python           build wheels in a stage with gcc → slim runtime.
                    (`pip install --target` or a venv copied across.)
   Frontend         node build stage → nginx:alpine serving /usr/share/nginx/html

Stages are not just for size

Three other uses that are less well known:

   FROM base AS test
   RUN npm test                    ← a stage that only runs in CI:
                                      docker build --target test .

   FROM base AS lint
   RUN golangci-lint run           ← runs in PARALLEL with test,
                                      because the DAG has no edge
                                      between them

   FROM alpine AS certs            ← a tiny stage that exists only to
   RUN apk add ca-certificates       supply files to a scratch image
   …
   FROM scratch
   COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

--target builds only up to a named stage, and BuildKit skips everything the target does not depend on. So one Dockerfile can serve --target dev, --target test and the default production build, with no duplication.

You can also copy directly from an image without a stage:

   COPY --from=ghcr.io/org/tool:1.4 /usr/bin/tool /usr/bin/tool

Cache mounts — the biggest single win

Chapter 6’s step 4 explained the problem: a RUN layer is the diff of the whole filesystem afterwards. So a package manager’s cache directory becomes part of your image, and is thrown away between builds anyway because the layer is discarded on a cache miss.

RUN --mount=type=cache attaches a directory that persists across builds and is not part of any layer.

   RUN --mount=type=cache,target=/root/.npm \
       npm ci

   RUN --mount=type=cache,target=/root/.cache/pip \
       pip install -r requirements.txt

   RUN --mount=type=cache,target=/go/pkg/mod \
       --mount=type=cache,target=/root/.cache/go-build \
       go build -o /app ./cmd

   RUN --mount=type=cache,target=/root/.cargo/registry \
       --mount=type=cache,target=/app/target \
       cargo build --release

   RUN --mount=type=cache,target=/root/.m2 \
       mvn -o package

The effect is large and worth quantifying, because it changes how you think about the ordering rule:

                              without cache mount    with cache mount

   npm ci, cache HIT               0 s                    0 s
   npm ci, cache MISS            190 s                   14 s
   (one dependency changed)

Without it, adding a single dependency costs a full cold install. With it, the miss re-runs npm ci but every unchanged package comes from the mount. The ordering rule from chapter 5 still matters — a cache hit at 0 s always beats 14 s — but the cost of a miss drops by an order of magnitude, which is what makes CI predictable.

Two flags worth knowing:

   --mount=type=cache,target=/var/cache/apt,sharing=locked
        default is `shared` (concurrent builds share it). Use `locked`
        for tools that cannot handle concurrent access — apt is one.

   --mount=type=cache,target=/app/node_modules,uid=1000,gid=1000
        the mount is root-owned by default; set uid/gid when the RUN
        executes as a non-root USER.

For apt specifically you must also disable its auto-clean, which otherwise deletes the cache you just mounted:

   RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
       echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' \
         > /etc/apt/apt.conf.d/keep-cache
   RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
       --mount=type=cache,target=/var/lib/apt,sharing=locked \
       apt-get update && apt-get install -y --no-install-recommends curl

Note that with cache mounts you no longer need the rm -rf /var/lib/apt/lists/* cleanup — nothing was written into the layer to begin with.

Secret mounts — the correct way

Chapters 3 and 5 established the problem twice: a secret in a layer is in the image forever, and --build-arg is visible in docker history.

RUN --mount=type=secret mounts a file into the build step’s filesystem for the duration of that command only. It is never in a layer, never in the history, never in the image.

   # Dockerfile
   RUN --mount=type=secret,id=npm_token \
       NPM_TOKEN=$(cat /run/secrets/npm_token) \
       npm ci

   # command line
   docker build --secret id=npm_token,env=NPM_TOKEN .
   docker build --secret id=npm_token,src=./token.txt .

For private git dependencies, there is a dedicated SSH forwarding mount:

   RUN --mount=type=ssh \
       go mod download

   docker build --ssh default .        ← forwards your ssh-agent

Careful with one thing: the secret is available to the command, so the command must not write it anywhere persistent. cat /run/secrets/token > ~/.npmrc puts it straight back into a layer. Use it and discard it in the same process.

A third mount type is occasionally handy:

   RUN --mount=type=bind,source=.,target=/src \
       cp /src/config.json /app/

A bind mount gives read access to context files without copying them into a layer at all.

Making the cache survive CI

The local layer cache lives in /var/lib/docker. A CI runner that is a fresh container or VM per job has none of it, so every build is cold — which is exactly when you most want it warm.

BuildKit can export and import the cache from a registry.

   docker buildx build \
     --cache-from type=registry,ref=ghcr.io/org/myapp:buildcache \
     --cache-to   type=registry,ref=ghcr.io/org/myapp:buildcache,mode=max \
     -t ghcr.io/org/myapp:$SHA \
     --push .

   mode=min   (default) exports only the final image's layers.
              Small, and enough if you only rebuild the last steps.

   mode=max   exports cache for EVERY step, including intermediate
              stages. Larger, and much more effective for
              multi-stage builds — which is most of them.

Other cache backends, worth knowing when the registry option is awkward:

   type=gha              GitHub Actions cache. Native, no registry
                         needed, 10 GB limit per repo, evicted by LRU.
   type=s3 / type=azblob object storage. Good for large monorepos.
   type=local            a directory. Useful with a self-hosted runner
                         that has a persistent volume.
   type=inline           embeds the cache metadata IN the pushed image.
                         Zero extra infrastructure, but mode=min only.

The realistic effect on a CI pipeline:

   cold, no cache export       8 min 20 s
   registry cache, mode=min    3 min 10 s
   registry cache, mode=max      55 s

One caveat: cache mounts are not exported. type=cache mounts are local to the builder. On ephemeral runners you want both — registry cache for layers, and a persistent builder or a mounted volume if you want cache mounts too. This is one of the strongest arguments for a long-lived self-hosted builder over fresh runners.

Multi-platform builds

Chapter 3’s exec format error has one fix:

   docker buildx build \
     --platform linux/amd64,linux/arm64 \
     -t ghcr.io/org/myapp:v2 --push .

buildx builds both and pushes a manifest list. Two ways it can do the work:

   EMULATION (QEMU)          one machine builds both. Simple.
                             The foreign architecture is emulated,
                             and it is SLOW — 3-10x for compilation.
                             Fine for interpreted languages, painful
                             for Rust or C++.

   NATIVE NODES              a builder with a real arm64 machine and
                             a real amd64 machine, each building its
                             own platform in parallel.
                             docker buildx create --append --node …
                             Much faster; needs the machines.

For compiled languages there is a third option that beats both: cross-compile in a native build stage. Go and Rust do this well.

   FROM --platform=$BUILDPLATFORM golang:1.22 AS build
   ARG TARGETOS TARGETARCH
   RUN GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /app ./cmd

$BUILDPLATFORM (where the build runs) and $TARGETPLATFORM / $TARGETOS / $TARGETARCH (what is being produced) are supplied automatically by BuildKit. The compiler runs natively at full speed and emits a foreign binary — no emulation anywhere.

Note that --platform on buildx build requires --push (or --load for a single platform), because the local image store historically could not hold a manifest list. If you get “docker exporter does not currently support exporting manifest lists”, that is this.

Builders

buildx builds run in a builder instance, and the default one has limits.

   docker buildx create --name mybuilder --driver docker-container --use
   docker buildx inspect --bootstrap

   driver: docker           the built-in builder. Cannot do multi-platform
                            or advanced cache export.
   driver: docker-container BuildKit in its own container. Full features.
                            Its cache lives in a volume and survives.
   driver: kubernetes       BuildKit pods in a cluster. Shared team cache,
                            scales, and is the right answer for a large org.
   driver: remote           connect to a BuildKit daemon you run.

docker buildx du shows cache usage and docker buildx prune --filter until=168h trims it. An unbounded BuildKit cache on a build server is a common cause of a full disk; chapter 13 covers it with the rest of pruning.

Reproducibility

Builds are not reproducible by default. The main sources of drift:

   1. floating base tags       → pin by digest (chapter 3)
   2. `apt-get install curl`   → pin versions, or accept the drift
      resolving differently
   3. timestamps in layers     → SOURCE_DATE_EPOCH=0, plus
                                 `--output type=image,rewrite-timestamp=true`
   4. non-deterministic tools  → the hard one. Some compilers and
                                 bundlers embed paths or times.

Full bit-for-bit reproducibility is achievable and genuinely difficult; if you need it, Nix and Bazel are built for it and Docker is not. What most teams actually need is weaker and much easier: the ability to rebuild the same image from the same commit, which digest-pinned bases plus a lockfile gets you.

What to take from this chapter

Multi-stage builds work because only the final stage’s layers become the image. A Go service goes from ~840 MB to ~11 MB. Use --target to serve dev, test and production builds from one Dockerfile — and independent stages run in parallel.
RUN --mount=type=cache keeps a package manager’s cache outside every layer and across builds. A cache-missing npm ci drops from ~190 s to ~14 s. Use sharing=locked for apt, and uid=/gid= when the step runs as non-root.
RUN --mount=type=secret is the only correct way to use a credential in a build — never in a layer, never in docker history. --mount=type=ssh forwards your agent for private git dependencies. The command must not write the secret anywhere persistent.
Export the cache to a registry (--cache-to type=registry,mode=max) so ephemeral CI runners start warm. mode=max covers intermediate stages and is what multi-stage builds need. Cache mounts are not exported — that needs a persistent builder.
Multi-platform builds need buildx and produce a manifest list. QEMU emulation is 3–10× slower for compilation; cross-compiling in a --platform=$BUILDPLATFORM stage using $TARGETARCH avoids emulation entirely.
Use a docker-container builder for anything beyond the basics, and prune it — an unbounded BuildKit cache fills disks.
Docker gives you repeatable, not reproducible. Pin base digests and lockfiles; if you truly need bit-for-bit, that is Nix or Bazel territory.

Next: running containers — the lifecycle, PID 1 and signal handling, exit codes, and why your container takes ten seconds to stop.

Written by Sepuri Sai Krishna