3. Images, layers and registries

This is the cleverest part of Docker, and understanding it turns several mysteries into arithmetic — why your image is 1.4 GB when your app is 20 MB, why deleting secrets from an image does not remove them, and why a one-line code change re-uploads 3 MB instead of 400.

An image is not a disk snapshot

The natural guess is that an image is a .tar of a filesystem. It is not. It is a stack of diffs.

   layer 4   COPY app.jar /app/       + /app/app.jar               18 MB
   layer 3   RUN pip install -r …     + /usr/lib/python…/*        180 MB
   layer 2   RUN apt-get install …    + /usr/bin/*, /usr/lib/*     90 MB
   layer 1   FROM debian:12           + the whole base filesystem  74 MB
             ────────────────────────────────────────────────────────
                                              image size          362 MB

Each layer records only what changed relative to everything below it. To produce the filesystem the container sees, the layers are stacked and merged.

That merging is done by a union filesystem, and on modern Linux that means overlayfs, built into the kernel.

Overlayfs, concretely

Overlayfs takes a list of read-only directories and one writable directory, and presents a single merged view.

   mount -t overlay overlay              \
     -o lowerdir=/l4:/l3:/l2:/l1,        ← image layers, read-only
        upperdir=/containers/abc/diff,   ← the container's writable layer
        workdir=/containers/abc/work     ← overlayfs scratch space
     /containers/abc/merged              ← what the process sees as "/"

The rules are simple:

Reading a file: search top-down through upperdir, then each lowerdir. First match wins.
Listing a directory: entries from all layers are merged into one listing.
Writing to an existing file: this is the interesting case.

Copy-up

When a process modifies a file that lives in a read-only lower layer, overlayfs cannot edit it in place. So it does a copy-up: the entire file is copied to the writable layer first, and the write applies there.

   container opens /var/lib/db/data.db  (a 4 GB file in layer 2)
   and writes ONE BYTE
        ↓
   overlayfs copies all 4 GB into upperdir
        ↓
   then applies the 1-byte write

The write appears to take several minutes. Nothing is broken; you are watching a 4 GB copy.

This is why databases in containers must use volumes, not the container filesystem — not merely for persistence, but because the first write to any large pre-existing file pays a full copy. A volume is a bind mount that bypasses overlayfs entirely, so there is no copy-up at all.

Whiteouts — and why deleting makes images bigger

You cannot delete a file from a read-only layer. So when a container (or a build step) deletes a file that exists below, overlayfs creates a whiteout: a special marker in the upper layer meaning “pretend this is not here”.

The consequence is the single most misunderstood thing about images:

   RUN curl -O https://…/sdk.tar.gz     layer 3:  + 400 MB
   RUN tar xf sdk.tar.gz                layer 4:  + 380 MB
   RUN rm sdk.tar.gz                    layer 5:  + a whiteout marker

   final image size:  400 + 380 + ~0  =  780 MB

The rm did not remove anything. Layer 3 still contains the 400 MB tarball, byte for byte. It is still downloaded on every pull. It is still on disk. The rm only added a note saying “do not show this”.

And the security version of the same fact:

   RUN cp /secrets/deploy_key /root/.ssh/id_rsa && \
       git clone git@internal:repo && \
       rm /root/.ssh/id_rsa                          ← one layer

Written as one RUN, this is fine — the key is created and removed inside a single layer, and only the layer’s end state is captured. Written as three separate RUN lines, the key is permanently embedded in a layer, and anyone who pulls the image can extract it with docker save and tar. Deleting a secret in a later layer does not remove it. The only fixes are to keep the whole lifecycle in one layer, use a multi-stage build (chapter 7), or use BuildKit secret mounts (chapter 7, and the right answer).

Content addressing

Every layer is identified by the SHA-256 hash of its contents:

   sha256:8f3a2b1c9d…    ← this IS the layer's name

Not a version number, not a path — the hash. Which gives you three properties for free:

Deduplication. Two images built on python:3.12-slim reference the identical base layer digest. It is stored once on disk and transferred once over the network. Fifty microservices on the same base cost you one base.

Verification. After pulling, the client hashes what it received. If it does not match, the data is wrong or tampered with, and the pull fails. Integrity is not bolted on; it is how the addressing works.

Immutability. A digest cannot change meaning. python@sha256:abc… is exactly one artifact, forever, on every machine, in every registry.

That last point sets up the most important operational distinction in this chapter.

Tags are mutable, digests are not

   nginx:1.25              a TAG.    A pointer. Can be moved at any time,
                                     by whoever can push. Says nothing
                                     about what you will actually get.

   nginx@sha256:9f8a…      a DIGEST. Exactly one image. Forever.

nginx:1.25 today and nginx:1.25 in six weeks can be different images — the publisher rebuilt it with security patches, which is good, and also means your “pinned” version is not pinned. :latest is worse: it is just a tag named “latest”, with no special meaning to Docker whatsoever, and it commonly points at something months old.

For anything reproducible — production deployments, CI base images, compliance — use digests. They are ugly and they are correct.

The argument gets sharper still under GitOps, where the whole claim is that a git repository describes exactly what is running. A mutable tag in that repository breaks the claim quietly: nothing in git changed, and what runs did. The Argo CD guide works through the awkward consequence — something has to write the new tag into git, and every way of doing it has a cost — but the starting point is the same as here. A tag is a pointer; only a digest is an identity.

What an image really is

Pull an image and you get three kinds of object:

   1. MANIFEST      a small JSON document:
                      - which config blob
                      - which layer blobs, in order
                      - the media type and size of each

   2. CONFIG BLOB   JSON: the default CMD, ENTRYPOINT, ENV, USER,
                    WORKDIR, exposed ports, labels, architecture,
                    and the ordered list of layer diff_ids.
                    ** The image's SHA is the hash of THIS file. **

   3. LAYER BLOBS   the actual gzipped tarballs of filesystem diffs.
                    Usually all the bytes.

So an “image ID” is the hash of a small JSON config file, which references layers by hash. Everything is a hash pointing at hashes — a Merkle tree, the same structure git uses for commits.

That is also why docker images shows an image ID that appears nowhere in the registry: registries address by manifest digest (what you get with @sha256:), while the local ID is the config digest. Different hashes of different objects for the same image. Confusing, harmless, and worth knowing before it costs you an hour.

Multi-arch: a manifest of manifests

docker pull nginx works on an Apple Silicon laptop and on an x86 server, from the same tag. That is a manifest list (or “image index”):

   nginx:1.25  →  manifest list
                    ├─ linux/amd64    → manifest → config + layers
                    ├─ linux/arm64    → manifest → config + layers
                    └─ linux/arm/v7   → manifest → config + layers

The client says which platform it wants; the registry returns the matching manifest. This is why building on an M-series Mac and deploying to x86 without --platform produces:

   exec /app/server: exec format error

The image is fine. It is an arm64 binary on an amd64 kernel. Chapter 14 covers docker buildx build --platform linux/amd64,linux/arm64.

Storage on disk

   /var/lib/docker/
     overlay2/
       <layer-id>/diff/     the extracted files for that layer
       <layer-id>/link      a short name used in the mount options
       l/                   short symlinks (overlayfs mount options
                            have a page-size limit; this is the workaround)
     image/overlay2/
       imagedb/             config blobs
       layerdb/             layer metadata and the chain of parents
     containers/<id>/       config, the writable layer, and the log file
     volumes/               named volumes

Two practical notes.

Layers are stored extracted, not as tarballs. They are gzipped in transit and on the registry, expanded on disk. So “the image is 200 MB” (registry, compressed) and “it takes 550 MB in /var/lib/docker” (local, extracted) are both true and routinely differ by 2–3×.

docker system df is the command that matters. Reclaimable space accumulates in three places — dangling images from rebuilds, stopped containers with writable layers, and the BuildKit cache — and it is the most common cause of a CI machine dying at 3 a.m. with no space left on device. Chapter 13 covers pruning properly.

Registries — the distribution half

A registry is smaller than people expect: an HTTP API over content-addressed blob storage, standardized as the OCI Distribution Spec. Roughly:

   GET  /v2/                                        does it speak the API
   GET  /v2/<name>/manifests/<tag-or-digest>        fetch a manifest
   GET  /v2/<name>/blobs/<digest>                   fetch a layer
   POST /v2/<name>/blobs/uploads/                   start an upload
   PUT  /v2/<name>/manifests/<tag>                  publish

Push: only what is missing

The push protocol’s efficiency comes from one step. Before uploading a blob, the client asks whether the registry already has that digest:

   HEAD /v2/myapp/blobs/sha256:8f3a…
     → 200  registry already has it. SKIP. Nothing is uploaded.
     → 404  upload it.

So pushing a rebuilt image where only the top layer changed uploads only that layer. A 400 MB image with a 3 MB code change is a 3 MB push. This — plus cross-repository blob mounting, which lets a registry link a blob it already stores under another repository instead of receiving it again — is why a registry hosting a thousand images built on the same five bases stores those bases once.

Pull: parallel, then sequential

   1. fetch the manifest list, pick your platform
   2. fetch the manifest, then the config blob
   3. download the missing layers  ← in parallel, 3 at a time by default
   4. extract them                 ← in order, sequentially

Step 4 is why pull time is not proportional to download speed alone. Decompressing a 400 MB layer is single-threaded and CPU-bound, and on a busy node it can dominate. It is also why many small layers beat one huge layer for pull latency, up to a point (the limit is 127 layers; in practice stay well under 25).

Rate limits and mirrors

Docker Hub applies pull rate limits per IP for anonymous users. A CI fleet behind one NAT gateway is one IP, and it will hit the limit, usually on the busiest day. Two fixes, in order of preference: pull through a registry mirror or pull-through cache, and host your base images in your own registry. Both also protect you from the upstream image being deleted, which has happened at scale before.

What to take from this chapter

An image is a stack of filesystem diffs, not a disk snapshot. overlayfs merges read-only lower layers plus one writable upper layer into what the process sees as /.
Copy-up: modifying a file from a lower layer copies the whole file up first. One byte written to a 4 GB file copies 4 GB. This is a reason — beyond persistence — that databases need volumes, which bypass overlayfs.
Deleting a file in a later layer does not remove it. It writes a whiteout marker. The bytes are still downloaded and still on disk — so a secret added in one layer and deleted in another is still in the image. Fix it with one RUN, a multi-stage build, or BuildKit secret mounts.
Everything is content-addressed by SHA-256, giving deduplication, verification and immutability for free.
Tags are mutable pointers; digests are not. nginx:1.25 can change under you; nginx@sha256:… cannot. Pin digests for anything that must be reproducible. :latest has no special meaning.
An image is a manifest → a config blob (the image ID is this file’s hash) → layer blobs. Multi-arch adds a manifest list on top; a mismatch gives you exec format error.
Layers ship compressed and are stored extracted — the registry number and the disk number differ by 2–3×. Watch docker system df.
A registry is HTTP over content-addressed blobs. Push HEADs each digest first and uploads only what is missing, so a code-only rebuild pushes megabytes, not hundreds.

Next: Docker’s own architecture — the CLI, the daemon, containerd, the shim, and runc, and what Kubernetes actually removed in 2020.

Written by Sepuri Sai Krishna