8. Running containers
Docker · 8. Running containersBuilding an image is a batch job that either works or does not. Running one is where the subtle problems live, and almost all of them come back to the same thing: your process is PID 1, and PID 1 on Linux is a special job that your application was not written to do.
The lifecycle
Section titled “The lifecycle”docker run is not one operation. It is create then start, and separating them explains several behaviours.
docker create │ ▼ CREATED ── the writable layer exists, the config is │ recorded, nothing is running docker start │ ▼ RUNNING ── the process exists │ │ │ docker pause│ │ │docker kill / the process exits ▼ │ ▼ PAUSED │ EXITED ── the process is gone. The writable │ │ │ layer, the logs and the exit code docker unpause │ │ REMAIN until `docker rm`. └────┘ │ ▼ docker rm REMOVED ── writable layer deletedThree consequences:
An exited container still exists and still costs disk. Its writable layer, its logs and its config are all retained. This is deliberate — it is how you inspect a crash — and it is why docker ps -a on a long-lived machine shows hundreds of dead containers holding gigabytes. docker run --rm cleans up automatically, which is right for one-off commands and wrong for anything you might need to debug.
docker pause uses the cgroup freezer, not a signal. The processes are frozen mid-instruction; they cannot react, and they do not know time passed. TCP connections stay open and time out from the other end.
Restarting a container reuses the same writable layer. docker restart keeps filesystem changes; docker rm followed by docker run does not. That difference has surprised people debugging “why did my data survive last time”.
PID 1 — the problem
Section titled “PID 1 — the problem”Chapter 2 introduced this; here is the full version, because it costs more engineering time than anything else in Docker.
When you run docker stop:
1. Docker sends SIGTERM to PID 1 in the container (or whatever STOPSIGNAL says) 2. Docker waits — 10 seconds by default 3. If the container is still running, Docker sends SIGKILLThat is a reasonable, graceful design. It fails in three distinct ways.
Failure 1 — shell form makes sh your PID 1
Section titled “Failure 1 — shell form makes sh your PID 1” ENTRYPOINT node server.js → /bin/sh -c "node server.js"
PID 1: /bin/sh PID 7: node server.jsdocker stop signals PID 1, which is sh. sh does not forward signals to its children. It waits for node and exits when node exits. So:
SIGTERM → sh → ignored ... 10 seconds ... SIGKILL → the whole cgroup → node dies instantly, mid-requestEvery stop takes exactly 10 seconds. Every deploy drops in-flight requests. The exit code is 137. Nothing logs an explanation.
Fix: exec form. ENTRYPOINT ["node", "server.js"].
Failure 2 — an entrypoint script without exec
Section titled “Failure 2 — an entrypoint script without exec”Very common, because entrypoint scripts are genuinely useful (waiting for a dependency, expanding environment variables, running migrations):
#!/bin/sh ./wait-for-db.sh node server.js ← WRONG. The script is PID 1; node is its child. Same problem. #!/bin/sh ./wait-for-db.sh exec node server.js ← RIGHT. `exec` REPLACES the shell process with node. node becomes PID 1 itself.exec is the whole fix, and it is one word. When the entrypoint script must also handle arguments passed to docker run:
#!/bin/sh set -e ./wait-for-db.sh exec "$@" ← runs whatever CMD/arguments were given
# Dockerfile ENTRYPOINT ["/entrypoint.sh"] CMD ["node", "server.js"]This pattern — script does setup, then exec "$@" — is the standard one, and it is what the official Postgres, MySQL and Redis images all do.
Failure 3 — the application ignores SIGTERM
Section titled “Failure 3 — the application ignores SIGTERM”Chapter 2’s rule: PID 1 gets no default signal handlers. A normal process that does not handle SIGTERM is still killed by the kernel’s default action. PID 1 is not. The signal arrives and is discarded.
So an application with no SIGTERM handler — which is most applications, because on a normal machine it never mattered — hangs for the full 10 seconds.
The fix is in the application, and it is worth writing properly because it is the same code that makes rolling deploys work:
process.on('SIGTERM', () => { server.close(() => { // 1. stop accepting new connections db.end(); // 2. finish in-flight requests process.exit(0); // 3. exit cleanly }); });Every language has the equivalent: Python’s signal.signal(signal.SIGTERM, …), Go’s signal.Notify with http.Server.Shutdown, Java’s Runtime.addShutdownHook, .NET’s IHostApplicationLifetime.
If you cannot change the application, --stop-timeout at least stops it wasting 10 seconds:
docker run --stop-timeout 2 myappZombies, and --init
Section titled “Zombies, and --init”The other half of PID 1’s job: reaping. When a process’s parent dies, the kernel re-parents it to PID 1, which is expected to call wait() and clear the entry from the process table. An entry that is never cleared is a zombie — no memory, no CPU, just a row in the table holding a PID.
Applications that spawn subprocesses — running ffmpeg, shelling out to git, a Python worker using subprocess — will orphan some of them. Under normal Linux, the real init reaps them and nobody notices. As PID 1 in a container, your application must do it, and it does not.
docker exec c ps aux | grep defunct | wc -l → 4 after an hour → 8000 after a week → then: fork: Cannot allocate memoryThe fix is a real init as PID 1:
docker run --init myappThis inserts tini, a ~10 KB init, as PID 1. It reaps zombies and forwards signals correctly to your process, which conveniently also fixes failure 1 above. In Compose it is init: true. In Kubernetes there is no --init, so you either bake tini into the image (ENTRYPOINT ["/sbin/tini", "--", "node", "server.js"]) or use shareProcessNamespace, where the pause container reaps.
Use --init if your application spawns subprocesses. If it does not, it is harmless and still fixes signal forwarding, so it is a reasonable default.
Exit codes
Section titled “Exit codes”The exit code tells you a lot, and the encoding is standard Unix rather than Docker-specific.
0 clean exit 1 application error (the app's own choice) 2 misuse of a shell builtin
125 DOCKER ITSELF failed — a bad flag, a bad option. The container never started. 126 the command was found but is NOT EXECUTABLE (missing +x, or a script with CRLF line endings, or a shebang pointing at a shell the image does not have) 127 the command was NOT FOUND (typo, wrong path, or — very common — a binary built against glibc running on Alpine, where the dynamic loader is missing and the error is misleadingly "not found")
128+n killed by signal n: 130 128 + 2 SIGINT — Ctrl-C 137 128 + 9 SIGKILL — `docker kill`, OR the OOM killer, OR the 10-second stop timeout expiring 139 128 + 11 SIGSEGV — segfault 143 128 + 15 SIGTERM — a clean stop, handled properlyThe distinction worth memorizing:
143 the app received SIGTERM and exited. Correct behaviour. 137 the app was SIGKILLed. Either it ignored SIGTERM (fix the app), or it ran out of memory.To tell those two 137s apart:
docker inspect --format '{{.State.OOMKilled}}' <container> → true memory limit hit → false it ignored SIGTERM, or was killed explicitlyexec format error is worth adding here as the fourth common one: an arm64 binary on amd64, or vice versa — chapter 3’s platform mismatch, not a permissions problem.
Restart policies
Section titled “Restart policies” --restart no (default) never restart --restart on-failure[:N] restart only on non-zero exit, at most N times --restart always always restart, INCLUDING after the daemon restarts and after a reboot --restart unless-stopped like always, but if you explicitly stopped it, it stays stopped across rebootsunless-stopped is almost always what you want for a long-running service on a single machine. always will resurrect a container you deliberately stopped when the daemon next starts, which is rarely intended.
Docker applies exponential backoff — 100 ms, 200 ms, 400 ms, doubling to a maximum of one minute — so a container in a crash loop does not spin the CPU. This is the same shape as Kubernetes’ CrashLoopBackOff, and the diagnosis is the same: the logs from the previous attempt are what you need.
docker logs <container> the current/last run docker inspect <container> | jq '.[0].State' RestartCount, ExitCode, Error, OOMKilled, FinishedAtA container restarting cleanly every 30 seconds with exit code 0 usually means the main process is daemonizing — forking into the background and letting the foreground process exit. Docker sees the foreground process end and considers the container finished. The main process must run in the foreground: nginx -g "daemon off;", apache2ctl -DFOREGROUND, postgres rather than pg_ctl start.
exec, attach and logs
Section titled “exec, attach and logs” docker exec -it c sh starts a NEW process inside the existing namespaces. Does not disturb PID 1. This is what you want 95% of the time.
docker attach c connects your terminal to PID 1's stdio. Ctrl-C sends SIGINT TO YOUR APP and usually kills the container. Detach with Ctrl-P Ctrl-Q instead.
docker logs -f c reads the log file the daemon captured from PID 1's stdout/stderr.Two things people trip over.
docker exec needs a shell in the image. A distroless or scratch image has no sh, so docker exec -it c sh fails with “executable file not found”. Options: docker debug in recent Docker Desktop, or docker run --pidns=container:<id> --network=container:<id> --cap-add SYS_PTRACE -it nicolaka/netshoot to join the target’s namespaces from a container that does have tools. That trick — a debug container sharing the target’s namespaces — is the general answer, and it is the same idea as Kubernetes’ ephemeral containers.
Logs come from stdout and stderr, and only from PID 1’s. If your application writes to a file inside the container, docker logs shows nothing, the file grows until it fills the writable layer, and it disappears on docker rm. Log to stdout. This is not a Docker preference so much as the only thing that works with any log collector.
The json-file driver’s log is at /var/lib/docker/containers/<id>/<id>-json.log and it is unbounded by default — a chatty service can fill a disk in days. Fix it globally in /etc/docker/daemon.json:
{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }That is chapter 13’s territory, but it is worth doing on day one of any machine that will run containers for more than a week.
Useful run flags, grouped
Section titled “Useful run flags, grouped” RESOURCES --memory 512m --memory-swap 512m (equal = no swap) --cpus 1.5 --pids-limit 200
LIFECYCLE --rm delete on exit --restart unless-stopped --init tini as PID 1 --stop-timeout 30
IDENTITY --user 1000:1000 override the image's USER --name api a stable name; also the DNS name on a user network
FILESYSTEM --read-only root fs read-only --tmpfs /tmp with a writable /tmp -v data:/var/lib/app a named volume
SECURITY --cap-drop ALL --cap-add NET_BIND_SERVICE --security-opt no-new-privileges
ENV -e KEY=value --env-file .env--read-only --tmpfs /tmp is a strong and underused pair: the application cannot modify its own filesystem, which removes a large class of persistence-based attacks and also catches, at development time, any accidental reliance on writing to the container filesystem. Chapter 11 goes further.
What to take from this chapter
Section titled “What to take from this chapter”- The lifecycle is created → running → exited → removed, and an exited container still holds its writable layer, logs and exit code until
docker rm. That is a feature for debugging and a disk leak on servers. - PID 1 is the recurring problem. Three separate failures produce the same 10-second stop and dropped requests: shell form in ENTRYPOINT, an entrypoint script without
exec, and an application with no SIGTERM handler (PID 1 gets no default handlers). - The fixes, in order: exec form,
exec "$@"at the end of entrypoint scripts, and a real signal handler in the app that stops accepting connections and drains. - PID 1 must also reap orphans. Applications that spawn subprocesses accumulate zombies until the PID limit.
--initinsertstini, which reaps and forwards signals — a reasonable default. - Exit codes: 125 Docker’s own failure, 126 not executable, 127 not found (or a glibc binary on musl), 137 SIGKILL — check
.State.OOMKilledto tell “out of memory” from “ignored SIGTERM” — and 143 a clean SIGTERM shutdown. --restart unless-stoppedfor services;alwaysresurrects containers you deliberately stopped. Restart backoff is exponential to one minute.- A container that exits 0 immediately is usually daemonizing. Run the process in the foreground.
docker execjoins the namespaces with a new process;docker attachconnects to PID 1 and Ctrl-C kills it. For images with no shell, join the target’s namespaces from a debug container.- Log to stdout, and set
max-size/max-fileon the json-file driver — it is unbounded by default.
Next: storage — why the writable layer is not storage, the three kinds of mount, and the permission problem everyone hits once.