13. Tuning and operations

This chapter is the practical maintenance layer: the settings and habits that keep Docker from becoming a problem on machines that run it for months.

Image size

Size affects three things: pull time on every deploy and every scale-out, disk on every node, and attack surface.

   a naive Node image                     1.2 GB
   + -slim base instead of full           680 MB
   + multi-stage, prod deps only          210 MB
   + no build toolchain in final          180 MB
   ─────────────────────────────────────────────
   the same application                   180 MB

The levers, in the order that pays:

Multi-stage (chapter 7). Usually the largest single win, and for compiled languages it is enormous — a Go binary in scratch is 8–15 MB against 800+ MB for the build image.

A smaller base (chapter 5). -slim over full, distroless over -slim where you can live without a shell. Remember chapter 5’s warning about Alpine and musl.

One RUN for install-and-clean. Chapter 3’s whiteouts: cleanup in a later layer removes nothing.

Do not copy what you do not need. .dockerignore, and COPY dist/ ./dist rather than COPY . . in the final stage.

To find out where the bytes actually are:

   docker history myapp:latest       size per layer, with the instruction
   docker image inspect myapp | jq '.[0].RootFS.Layers | length'
   dive myapp:latest                 an interactive layer browser;
                                     shows exactly which files each
                                     layer added and flags wasted space

dive is worth installing. It routinely finds a 300 MB apt cache or a copied .git directory that nobody knew was there.

Build speed

Ranked by typical impact:

   1. Order instructions correctly (ch 5)      4 min → 8 s
   2. Cache mounts for package managers (ch 7) 190 s → 14 s on a miss
   3. .dockerignore                            20 s → 0.4 s of context
   4. Registry cache export for CI (ch 7)      8 min → 55 s cold
   5. Cross-compile instead of QEMU (ch 7)     6 min → 40 s multi-arch
   6. A persistent buildx builder              keeps 2 and its cache

Read the BuildKit output as chapter 6 described: the first line without CACHED is the cause of everything below it. For a deeper look:

   docker buildx build --progress=plain .       full output, no collapsing
   docker buildx du --verbose                   what the cache holds

The disk that fills up

This is the most common Docker operational failure, and it has one command:

   docker system df

   TYPE            TOTAL   ACTIVE  SIZE     RECLAIMABLE
   Images          142     8       48.2GB   41.9GB (86%)
   Containers      67      6       2.1GB    1.9GB  (90%)
   Local Volumes   38      4       112GB    98GB   (87%)
   Build Cache     891     0       27.4GB   27.4GB

Four separate accumulators, each with its own cause:

   IMAGES        every rebuild leaves the previous one dangling
                 (an untagged <none> image).
   CONTAINERS    exited containers keep their writable layer and logs
                 until `docker rm` (chapter 8).
   VOLUMES       anonymous volumes from `VOLUME` instructions, one per
                 `docker run`, forever (chapter 9).
   BUILD CACHE   BuildKit keeps everything until told otherwise (ch 7).

Cleaning, from safe to dangerous:

   docker image prune                 dangling (<none>) images only. Safe.
   docker container prune             all stopped containers.
   docker builder prune               build cache.
   docker system prune                the three above together.

   docker system prune -a             ALSO every image not used by a
                                      RUNNING container. Your entire
                                      local image library. Next build
                                      re-pulls everything.

   docker system prune -a --volumes   ALSO every unused volume.
                                      This deletes databases.

Use filters instead of blunt force:

   docker image prune -a --filter "until=168h"
   docker builder prune --filter "until=168h" --keep-storage=20gb
   docker container prune --filter "until=24h"

And configure BuildKit to bound itself rather than relying on cron:

   {
     "builder": {
       "gc": {
         "enabled": true,
         "defaultKeepStorage": "20GB"
       }
     }
   }

A weekly docker system prune -af --filter "until=168h" on build servers, plus builder GC, prevents almost all of these incidents. On production hosts, prefer image and container pruning with filters and leave volumes strictly alone.

Logs

Chapter 8 covered this and it belongs in any operations checklist because the default is genuinely dangerous:

The json-file log driver is unbounded by default. A container logging 5 MB/minute fills 200 GB in a month.

   {
     "log-driver": "json-file",
     "log-opts": { "max-size": "10m", "max-file": "3" }
   }

That is 30 MB per container, ceiling. It applies to newly created containers only, so existing ones need recreating.

Other drivers, briefly:

   json-file    default. `docker logs` works. Rotate it.
   local        Docker's own format. More efficient, rotates by
                default. `docker logs` works. A better default if
                nothing parses the raw files.
   journald     into systemd's journal. Host-level rotation and
                retention already solved. `docker logs` still works.
   fluentd / gelf / awslogs / splunk
                ship directly to a collector.
                ** `docker logs` STOPS WORKING ** with these — the
                daemon does not keep a local copy. Surprising during
                an incident.

The application side is the same one sentence from chapter 8: log to stdout, structured, and let the platform collect it. Log files inside a container are invisible to every collector and lost on removal.

Daemon settings worth setting on day one

   {
     "log-driver": "json-file",
     "log-opts": { "max-size": "10m", "max-file": "3" },

     "default-address-pools": [
       { "base": "172.20.0.0/16", "size": 24 }
     ],

     "live-restore": true,

     "builder": { "gc": { "enabled": true, "defaultKeepStorage": "20GB" } },

     "default-ulimits": {
       "nofile": { "Name": "nofile", "Hard": 65536, "Soft": 65536 }
     },

     "registry-mirrors": ["https://mirror.internal.example.com"]
   }

   default-address-pools   Docker's default 172.17-172.31 range collides
                           with corporate VPNs and cloud VPCs constantly.
                           The symptom is that a container can reach the
                           internet but not one internal subnet, and it
                           takes hours to find. Set the pool deliberately.

   live-restore            containers keep running across a daemon
                           restart. Chapter 4's shim makes this possible;
                           this flag turns it on. Not compatible with
                           Swarm mode.

   default-ulimits         the default file descriptor limit is low for
                           servers holding many connections.

   registry-mirrors        a pull-through cache. Chapter 3's Docker Hub
                           rate limits, solved.

Apply with systemctl reload docker for most keys; some need a full restart.

Resource limits in production

   --memory 512m       a hard ceiling. Exceeding it is an OOM kill.
   --memory-reservation 256m
                       a soft target. Under host pressure, the kernel
                       reclaims from containers above their reservation
                       first. Useful and underused.
   --memory-swap 512m  equal to --memory means NO SWAP. Recommended:
                       swapping in a container produces latency that
                       is very hard to diagnose.
   --cpus 1.5          a quota. Throttles, silently (chapter 2).
   --cpu-shares 1024   a relative weight under contention only.
                       Not a cap. Better for latency-sensitive work.
   --pids-limit 200    fork bomb protection.
   --ulimit nofile=65536:65536

The guidance that carries over from the Kubernetes guide and applies identically here: always set a memory limit; think hard before setting a CPU limit. A memory limit converts an unbounded leak into one dead container instead of a dead machine. A CPU limit converts a burst of work into silent latency, with nothing in any log to explain it. For latency-sensitive services, --cpu-shares gives you fair scheduling under contention without capping bursts.

Monitoring

   docker stats                 live CPU, memory, network, block I/O
   docker stats --no-stream     one snapshot, scriptable
   docker events                a stream of lifecycle events —
                                excellent for understanding what is
                                restarting things
   docker inspect <c> | jq '.[0].State'
                                RestartCount, ExitCode, OOMKilled

For real monitoring, cAdvisor exposes per-container metrics to Prometheus, and it is what most container dashboards are built on — see the Prometheus guide for how those metrics are scraped, stored and queried. The four signals that matter:

   container_memory_working_set_bytes / the limit   → approaching OOM
   container_cpu_cfs_throttled_periods_total        → CPU throttling.
                                                      Almost nobody
                                                      graphs this, and
                                                      it explains a lot
                                                      of mystery latency.
   container_fs_usage_bytes                         → writable layer growth
   restart counts                                   → crash loops

The throttling metric deserves the emphasis. When a service is inexplicably slow at p99 and every application-level metric looks fine, throttled_periods is usually the answer.

Docker Desktop on macOS and Windows

Worth its own section because the performance complaints are constant and the cause is structural.

   Linux                        macOS / Windows

   containers run on the        containers run inside a LINUX VM
   host kernel directly         (Docker Desktop's VM, or WSL2)

   a bind mount is a bind       a bind mount crosses the VM boundary
   mount — zero cost            via a file-sharing protocol
                                (VirtioFS, gRPC-FUSE, or 9p)

That boundary crossing is the whole problem. A node_modules tree with 40,000 small files, or a Rails app doing thousands of stat calls per request, can be 10–50× slower through a bind mount than on the host filesystem.

Mitigations, in order of effect:

   1. Do not bind-mount huge dependency trees. Use chapter 9's
      anonymous-volume mask so node_modules/vendor/target stay
      inside the container.
   2. Use `docker compose watch` (chapter 12) — it syncs changed
      files instead of proxying every filesystem call.
   3. Enable VirtioFS in Docker Desktop settings. It is much faster
      than the older gRPC-FUSE.
   4. On Windows, keep the code inside the WSL2 filesystem
      (/home/you/project), not on /mnt/c. Crossing to the Windows
      filesystem is the single biggest slowdown there.
   5. Consider Colima, OrbStack or Rancher Desktop. OrbStack in
      particular is noticeably faster for filesystem-heavy work.

Also remember chapter 3: Apple Silicon builds arm64 by default. Add --platform linux/amd64 when the target is an x86 server, or build multi-arch.

What to take from this chapter

Image size: multi-stage first, then a smaller base, then one-RUN cleanup, then copying less. Use dive to find where the bytes actually went.
Build speed, in order: instruction ordering, cache mounts, .dockerignore, registry cache export, cross-compilation over QEMU. The first uncached line in the output is always the cause.
docker system df names the four accumulators — images, containers, volumes, build cache. Prune with --filter until= rather than blunt commands, and never -a --volumes on anything holding data. Turn on BuildKit GC in daemon.json.
The json-file log driver is unbounded by default — set max-size and max-file on day one; it only applies to newly created containers. Drivers that ship logs elsewhere break docker logs.
Day-one daemon.json: log rotation, default-address-pools (the default range collides with VPNs and VPCs and costs hours to diagnose), live-restore, builder GC, ulimits, and a registry mirror.
Always set a memory limit; think hard before a CPU limit. Memory limits turn a leak into one dead container; CPU limits turn a burst into silent latency. --cpu-shares gives fairness without a cap.
Graph container_cpu_cfs_throttled_periods_total. It explains mystery p99 latency more often than anything else on the list.
Docker Desktop is slow because bind mounts cross a VM boundary. Keep dependency trees inside the container, use compose watch, enable VirtioFS, and on Windows keep code in the WSL2 filesystem.

Next: the failure catalogue — every recurring Docker failure, what it looks like, and what causes it.

Written by Sepuri Sai Krishna