14. Failure modes

Everything in this chapter is something that will happen to you. They are grouped by where they show up, and each one names the chapter that explains the mechanism.

Build failures

The cache never hits

   LOOKS LIKE   every build re-runs `npm ci` / `pip install`, even
                when you only changed one source file.

   CAUSE        `COPY . .` before the install step. Any change to any
                file in the context invalidates it and everything below.
                (chapter 6)

   ALSO         a file the build does not use is changing — a .log from
                a test run, a .git object, a coverage report. `COPY . .`
                hashes ALL of it.

   FIX          copy the dependency manifest first, install, then copy
                source. Fix .dockerignore. (chapter 5)

The build works locally and not in CI

   LOOKS LIKE   passes on a laptop, fails on the runner.

   CAUSE 1      the runner is cold — no layer cache. If the build
                depended on a cached step that "always worked", the
                cold path was never tested.
   CAUSE 2      DOCKER_BUILDKIT=0 somewhere in the CI config, so
                cache mounts and secret mounts silently do nothing.
   CAUSE 3      .dockerignore excludes something the build needs,
                and the laptop had it from a previous run.
   CAUSE 4      different architecture — an ARM laptop, an x86 runner.

   FIX          reproduce with `docker build --no-cache --pull`.
                That is the CI path.

apt-get 404s on an unchanged Dockerfile

   LOOKS LIKE   E: Failed to fetch … 404  Not Found

   CAUSE        `RUN apt-get update` and `RUN apt-get install` in
                SEPARATE instructions. The update layer is cached from
                weeks ago; the package index it wrote points at
                versions that have since been superseded and removed.

   FIX          one RUN: `apt-get update && apt-get install -y …`
                (chapter 5)

The image is enormous

   LOOKS LIKE   1.4 GB for a small service.

   CAUSE 1      no multi-stage — the compiler, headers and package
                manager shipped to production. (chapter 7)
   CAUSE 2      downloaded-then-deleted files. `rm` in a later layer
                removes NOTHING. (chapter 3)
   CAUSE 3      the package manager's cache is in a RUN layer.
                (chapter 7's cache mounts)
   CAUSE 4      `COPY . .` brought in .git and node_modules.

   FIX          `dive myimage` shows exactly which layer and which
                files. (chapter 13)

A secret is in the image

   LOOKS LIKE   nothing. That is the problem.

   CAUSE        --build-arg (visible in `docker history`), or a file
                COPYed in and deleted later (still in its layer), or
                a credential written to ~/.npmrc or ~/.netrc.

   CHECK        docker history --no-trunc myimage
                docker save myimage | tar -tv

   FIX          RUN --mount=type=secret. Then rotate the credential —
                if the image was pushed, assume it is public.
                (chapter 7)

Container will not start, or will not stay up

Exits immediately with code 0

   CAUSE        the main process daemonized — forked to the background
                and the foreground process exited. Docker saw PID 1
                finish and considered the container done.

   FIX          run in the foreground:
                  nginx -g "daemon off;"
                  apache2ctl -D FOREGROUND
                  postgres (not pg_ctl start)
                (chapter 8)

Exit code 127 — “not found” for a binary that exists

   CAUSE 1      a typo or wrong path. Check with:
                  docker run --rm --entrypoint ls myimage /app

   CAUSE 2      *** a dynamically linked glibc binary on Alpine. ***
                The binary is there. The DYNAMIC LOADER it names
                (/lib64/ld-linux-x86-64.so.2) is not, and the kernel
                reports that as "not found" — about the binary.
                Confirm with `ldd /app/binary` inside the container.

   FIX          build static (CGO_ENABLED=0 for Go), or use a
                glibc base (-slim), or install libc6-compat on Alpine.
                (chapter 5)

Exit code 126 — permission denied

   CAUSE 1      missing execute bit. COPY preserves host permissions.
   CAUSE 2      *** CRLF line endings on a shell script. ***
                The shebang becomes "#!/bin/sh\r", and there is no
                interpreter named "sh\r". The error mentions none of
                this.

   FIX          COPY --chmod=755 entrypoint.sh /
                and add `*.sh text eol=lf` to .gitattributes

exec format error

   CAUSE        architecture mismatch. An arm64 image on an amd64 host
                or vice versa — very common when building on Apple
                Silicon and deploying to x86.

   CHECK        docker image inspect myimage | grep Architecture

   FIX          --platform linux/amd64, or buildx multi-arch.
                (chapters 3 and 7)

CrashLoopBackOff-style restarting

   CHECK        docker logs <container>          the last run
                docker inspect <c> | jq '.[0].State'
                  → ExitCode, OOMKilled, RestartCount, Error

   137 + OOMKilled true   → raise the memory limit, or fix the leak
   137 + OOMKilled false  → it ignored SIGTERM (see below)
   1                      → application error; read the logs
   0 in a loop            → daemonizing

The container runs but misbehaves

docker stop always takes exactly 10 seconds

   THE most common Docker problem in production, and it looks like
   nothing — deploys are just slow and requests get dropped.

   CAUSE 1      shell-form ENTRYPOINT → /bin/sh is PID 1 and does not
                forward SIGTERM.
   CAUSE 2      an entrypoint script without `exec` — same shape.
   CAUSE 3      the app has no SIGTERM handler, and PID 1 gets no
                default handler from the kernel.

   CHECK        docker exec c ps -eo pid,comm
                → is PID 1 your app, or is it sh?

   FIX          exec form; `exec "$@"`; a real signal handler.
                Interim: --init, or --stop-timeout 2.
                (chapter 8)

Zombie processes accumulate

   LOOKS LIKE   `ps aux` full of <defunct>; eventually
                "fork: Cannot allocate memory"

   CAUSE        PID 1 must reap orphans. Your app does not.

   FIX          docker run --init  (tini)
                or ENTRYPOINT ["/sbin/tini","--","app"]
                (chapter 8)

The app is slow, and no metric explains it

   CAUSE        CPU throttling. A --cpus limit means "N ms of CPU per
                100 ms window, then frozen". Nothing dies. Nothing logs.

   CHECK        cat /sys/fs/cgroup/…/cpu.stat
                  → nr_throttled, throttled_usec
                or the Prometheus metric
                  container_cpu_cfs_throttled_periods_total

   FIX          raise or remove the CPU limit; use --cpu-shares for
                fairness instead of a cap. (chapters 2 and 13)

The JVM (or Node, or .NET) is OOM-killed with plenty of headroom

   CAUSE        the runtime sized its heap from /proc/meminfo, which
                reports the HOST's memory — cgroup limits do not
                change it. A 512 MB container on a 64 GB host plans
                for a 16 GB heap.

   FIX          -XX:MaxRAMPercentage=75
                node --max-old-space-size=384
                DOTNET_GCHeapHardLimit
                (chapter 2)

Writes disappear on restart

   CAUSE        the app writes to the container's own filesystem.
                The writable layer is deleted with the container.

   FIX          a volume. Also, `docker restart` keeps the layer while
                `rm` + `run` does not, which makes this look
                intermittent. (chapter 9)

Storage

Permission denied on a mounted directory

   CAUSE        UIDs are numbers. The host directory is owned by 1000;
                the container user is 999. No name matching happens.

   FIX          -u "$(id -u):$(id -g)"           (development)
                chown on the host                (server)
                a root entrypoint that chowns then `exec gosu`
                or use a named volume, which is seeded with the
                right ownership. (chapter 9)

   NOTE         Docker Desktop hides this. It appears the first time
                the setup runs on a Linux CI runner.

A rebuilt image serves the old files

   CAUSE        a named volume mounted over that directory. It was
                seeded from the image ONCE, on first use, and now
                masks the image entirely.

   FIX          docker volume rm <name>, or `docker compose down -v`
                if you are certain of what is in there. (chapter 9)

MODULE_NOT_FOUND after bind-mounting the source

   CAUSE        `-v $(pwd):/app` hides /app/node_modules that the
                image installed. Or a host node_modules built for a
                different OS/arch is now inside a Linux container.

   FIX          -v $(pwd):/app -v /app/node_modules
                (an anonymous volume at the more specific path wins)
                (chapter 9)

no space left on device

   CHECK        docker system df
                df -h /var/lib/docker

   FOUR CAUSES  dangling images from rebuilds
                exited containers holding writable layers
                anonymous volumes (one per run of a VOLUME image)
                an unbounded BuildKit cache
   PLUS         unrotated json-file logs — the sneaky fifth one.

   FIX          docker system prune --filter "until=168h"
                log rotation in daemon.json
                builder GC in daemon.json
                NOT `prune -a --volumes` unless you are certain.
                (chapter 13)

Networking

Connection refused to another container

   CAUSE 1      using `localhost`. Inside a container that means THAT
                container. Use the service/container name.
   CAUSE 2      on the default `bridge` network, which has NO DNS.
                Create a user-defined network (Compose does).
   CAUSE 3      the service is listening on 127.0.0.1 INSIDE its own
                container, so nothing outside it can connect.
                Bind to 0.0.0.0.
   CAUSE 4      different Compose projects → different networks.

   CHECK        docker exec c getent hosts db
                docker exec c nc -zv db 5432
                (chapter 10)

The port is not reachable from outside

   CAUSE 1      EXPOSE instead of -p. EXPOSE publishes nothing.
   CAUSE 2      published to 127.0.0.1 only.
   CAUSE 3      the app inside is bound to 127.0.0.1, so the DNAT
                rule delivers to a socket nothing is listening on.

   CHECK        docker port <container>
                docker exec c ss -lntp
                (chapter 10)

A port is open that your firewall says is closed

   CAUSE        Docker's DNAT rules are in iptables PREROUTING, which
                runs BEFORE the filter/INPUT chain that UFW and
                firewalld use. UFW is never consulted.

   FIX          -p 127.0.0.1:5432:5432 — always bind explicitly.
                This has exposed real databases to the internet.
                (chapter 10)

DNS fails intermittently

   CAUSE 1      a systemd-resolved host: /etc/resolv.conf points at
                127.0.0.53, which Docker strips, and containers fall
                back to 8.8.8.8 — so internal names stop resolving.
                Fix: "dns": [...] in daemon.json.
   CAUSE 2      Alpine/musl: parallel queries, different search-domain
                handling, no reliable TCP fallback for large responses.
                Fix: use a -slim base.
   (chapter 10)

Containers can reach the internet but not one internal subnet

   CAUSE        Docker's default 172.17–172.31 pools overlap a VPN or
                VPC range. Traffic for that subnet is routed into the
                bridge instead of out.

   FIX          "default-address-pools" in daemon.json. Set it before
                you need it — this one costs hours to diagnose.
                (chapter 13)

Production-only failures

It worked in staging

   Usual causes, in order:
     - a floating tag (:latest, or a rebuilt :1.25) resolved to a
       different image (chapter 3)
     - a cold layer cache exposed a step that had never actually run
     - different architecture
     - a memory limit that staging did not have
     - a bind mount in dev that is a real image path in prod, so
       nobody noticed the file was never COPYed in

Everything slows down at the same time on one host

   CHECK  docker stats
          df -h /var/lib/docker
          dmesg | grep -i oom
          throttled_periods per container

   USUAL  one container without a memory limit is causing host-wide
          reclaim; or the disk is full; or one container is eating
          all the CPU and there are no shares configured.

A pull fails with toomanyrequests

   CAUSE        Docker Hub anonymous rate limits, per IP. A CI fleet
                behind one NAT gateway is one IP.

   FIX          a registry mirror / pull-through cache, and host your
                base images yourself. (chapters 3 and 13)

The diagnostic order

When something is wrong and you do not know where to start:

   1. docker ps -a                 is it running? what exit code?
   2. docker logs --tail 100 <c>   what did it say?
   3. docker inspect <c> | jq '.[0].State'
                                   OOMKilled? RestartCount? Error?
   4. docker exec <c> ps -eo pid,comm
                                   is PID 1 your app or a shell?
   5. docker exec <c> ss -lntp     is it listening, and on what?
   6. docker system df             is the disk the problem?
   7. docker events                what keeps restarting it?

Steps 1–4 identify most problems in under a minute.

What to take from this chapter

• The 10-second stop is the most expensive routine failure. It is always PID 1: shell form, a missing exec, or no SIGTERM handler. Check with docker exec c ps -eo pid,comm — if PID 1 is sh, that is your answer.
• 127 is often not a typo — it is a glibc binary on musl, where the missing dynamic loader is reported as the binary being missing. 126 is often CRLF line endings in a shell script.
• 137 has two meanings. .State.OOMKilled distinguishes “out of memory” from “ignored SIGTERM”.
• Cache misses are COPY . . in the wrong place, or a stray file the context should have ignored. apt 404s on an unchanged Dockerfile are a split update/install.
• Permission denied on a mount compares UID numbers, not names — and Docker Desktop hides it until Linux CI finds it.
• Published ports bypass UFW. Always -p 127.0.0.1:port:port.
• default-address-pools collisions with a VPN or VPC produce “the internet works but this one subnet does not”, and are worth setting before you ever hit it.
• Disk fills from five places: dangling images, exited containers, anonymous volumes, BuildKit cache, and unrotated logs.
• Diagnostic order: ps -a → logs → inspect .State → ps inside the container. That is four commands and most of the answers.

Next: when to use Docker, when not, and what the honest alternatives are.