1. What Docker is and why it exists
Docker · 1. What Docker is and why it existsDocker is the most misunderstood tool in this collection, and the misunderstanding is specific: most people think Docker invented containers. It did not. Every kernel feature Docker uses existed before Docker did, some of them for a decade.
What Docker invented was something less glamorous and far more valuable. To see it, you have to start with the problem.
The problem
Section titled “The problem”You have an application. It runs on your laptop. It needs to run on a server.
That sentence hides an enormous amount of pain, and the pain has a shape:
YOUR LAPTOP THE SERVER
Python 3.11 Python 3.9 OpenSSL 3.0 OpenSSL 1.1.1 libpq 15 libpq 12 glibc 2.38 glibc 2.31 ImageMagick installed by ImageMagick installed by brew, 3 years ago apt, whatever version a PATH with 14 entries a PATH with 4 entries an env var you set once (and none of yours) and forgot aboutYour application does not depend on “Python”. It depends on that exact arrangement of files on disk. Change any of it and you get a stack trace, or worse, a subtle behavioural difference that shows up under load three weeks later.
The industry’s name for this was “works on my machine”, said with a particular kind of tiredness.
We had answers, and each one was almost good enough.
Config management — Puppet, Chef, Ansible. You write down how the machine should be set up, and a tool makes it so. This is a real improvement, and it is still useful. But it is convergent, not reproducible: you describe a destination and hope the machine gets there. Run it on a fresh box and a box that has been alive for two years and you can get two different results, because the tool only changes what it knows to change. And it is slow — minutes to bring up a machine.
Virtual machines. Ship the whole computer. This genuinely works: a VM image contains the kernel, the OS, your libraries, your app. Boot it anywhere, get exactly the same thing.
It is also the wrong shape for the job:
A VM What you actually wanted
┌────────────────────┐ │ your app 20 MB │ ┌────────────────────┐ ├────────────────────┤ │ your app 20 MB │ │ libraries 200 MB │ ├────────────────────┤ ├────────────────────┤ │ libraries 200 MB │ │ full OS 800 MB │ └────────────────────┘ ├────────────────────┤ │ a KERNEL 80 MB │ share the host's kernel ├────────────────────┤ │ virtual hardware │ no virtual hardware └────────────────────┘
boots in 30-60 s starts in 50 ms costs ~1 GB, ~512 MB RAM costs a few MB just to exist over the app itselfA VM gives you an entire simulated computer when all you needed was a reliable filesystem and some isolation. You pay for a kernel you do not need, hardware emulation you do not need, and a boot sequence you do not need — per copy of your app.
The isolation already existed
Section titled “The isolation already existed”Meanwhile, the Linux kernel had quietly grown everything required to run a process in its own private world:
2002 mount namespaces a process gets its own filesystem view 2006 cgroups (at Google) limit CPU/memory for a group of processes 2008 LXC packaged all of it into "Linux Containers" 2013 DockerBy 2008 you could already do, on plain Linux, what a container does. Google was running everything this way internally. LXC existed and worked.
Almost nobody used it. The commands were arcane, there was no standard way to describe a container, and — this is the important part — there was no way to give someone else the thing you had built. You could construct an isolated environment on your machine. You could not hand it to a colleague, or to a server, as a single artifact.
What Docker actually invented
Section titled “What Docker actually invented”Docker’s contribution was not isolation. It was three things that sound boring and changed the industry:
1. A packaging format that is layered and content-addressed. An image is not a disk snapshot. It is a stack of filesystem diffs, each identified by the hash of its contents. This is what makes images shareable at a practical cost: if you and I both build on ubuntu:22.04, we store and transfer that 70 MB once, not twice. Chapter 3 is entirely about this and it is the cleverest thing in the system.
2. A build file that is a script anyone can read. A Dockerfile is a sequence of steps. Run it and you get an image. Run it again and — mostly — you get the same image. Crucially, it is checked into the repository next to the code, so the environment and the application version together.
3. A registry, and a verb for using it. docker push and docker pull. A shared, HTTP-based place where images live, with the same UX as git push. This is the piece that turned a local capability into an ecosystem.
Put together, they produced a single sentence that was not previously true:
The thing you tested is bit-for-bit the thing that runs in production.
Not “an equivalent environment”. The same bytes, verified by hash.
The four things called “Docker”
Section titled “The four things called “Docker””A lot of confusion comes from one word covering four separate things. Pull them apart now and the rest of this guide is much easier.
1. THE IMAGE FORMAT now an open standard (OCI). Not owned by Docker. Kubernetes, Podman, containerd, every cloud — all use the same format.
2. THE BUILD TOOL "docker build" / BuildKit. Turns a Dockerfile into an image. Replaceable: Buildah, Kaniko, ko, Buildpacks all produce OCI images.
3. THE RUNTIME dockerd + containerd + runc. Actually starts the process. Replaceable: Kubernetes does not use dockerd at all any more.
4. THE DEVELOPER EXPERIENCE the CLI, Desktop, Compose. This is the part people love, and the part that is genuinely still best-in-class.This split explains something that confused everyone in 2020, when Kubernetes announced it was “deprecating Docker”. Kubernetes dropped #3 — the dockerd daemon — and kept #1 entirely. Your images did not change. Nothing you built stopped working. The layer being removed was a piece of plumbing that Kubernetes had been talking to through an adapter, and it went to containerd directly instead. Chapter 4 shows exactly what was removed and why.
What a container is, in one paragraph
Section titled “What a container is, in one paragraph”Full detail is chapter 2, but hold this from the start, because it prevents a whole class of wrong intuitions:
A container is a normal Linux process. It appears in ps on the host. kill -9 works on it. It is scheduled by the same kernel scheduler as everything else. What makes it a “container” is that when it was started, the kernel was told to give it a restricted view (namespaces — what it can see) and a restricted budget (cgroups — what it can use), and to root its filesystem at a directory assembled from image layers.
There is no “container” object in the Linux kernel. There is no container process type. Docker is a program that starts processes with unusual arguments.
This single fact explains a surprising amount:
- Containers start in milliseconds — because starting one is starting a process.
- Containers cannot run a different kernel — a Linux container needs a Linux kernel, always. (Docker Desktop on macOS and Windows quietly runs a Linux VM. Your “containers on Mac” are containers inside a virtual machine.)
- A container escape is a kernel bug — you are one kernel away from the host, not one hypervisor away. This is chapter 11.
docker rundoes not “boot” anything. There is no init, no login, no services. There is one process, and when it exits, the container is over.
The trade Docker makes
Section titled “The trade Docker makes”Every tool in this collection buys something and pays for it. Docker’s trade:
You gain a reproducible artifact and near-zero startup cost. You pay by sharing the host kernel, and by being forced to make your application stateless and configurable from outside.
The second half is the one that surprises people. Docker is not a neutral packaging format that accepts any application. It has strong opinions, and applications that disagree with them are painful to containerize:
- Write to disk and expect it to persist? The container’s filesystem is thrown away on restart. You must move to volumes (chapter 9).
- Read config from a file baked in at install time? You need environment variables or mounted config, because one image must run in dev, staging and production unchanged.
- Log to a file? Log to stdout instead; something else collects it.
- Run several services under one supervisor? Docker expects one process per container. You can fight this; you will lose slowly.
These constraints are not arbitrary. They are the same constraints that make an application easy to scale, easy to restart, and easy to schedule — which is why containerization and the move to twelve-factor style applications happened together. Docker did not just package apps. It pushed a generation of software into a shape that could be orchestrated, and that shape is what Kubernetes then assumed.
Where this guide is going
Section titled “Where this guide is going”Docker is one of the few tools where the concepts are small and the details are where everything goes wrong. Almost nobody is confused about what an image is. Almost everybody has, at some point, spent an afternoon on a build cache that would not invalidate, a container that would not stop, or a volume with the wrong permissions.
So this guide is weighted toward mechanism:
- Chapters 2–4 — what a container really is, how images are really stored, and what the pieces of Docker itself are.
- Chapters 5–7 — the Dockerfile, one build traced end to end, and the caching model. Chapter 5 is the one to read twice.
- Chapters 8–11 — running, storing, networking, securing.
- Chapters 12–16 — Compose, tuning, the failure catalogue, when to use something else, and a reference sheet.
If you have read the Kubernetes guide, this one sits directly underneath it. Kubernetes chapter 8 described namespaces, cgroups and image layers in a paragraph each because they were not the subject. Here they are the subject.
What to take from this chapter
Section titled “What to take from this chapter”- Docker did not invent containers. Namespaces, cgroups and LXC all predate it. What it invented was a layered content-addressed image format, a readable build file, and a registry — packaging and distribution, not isolation.
- The problem it solved was “works on my machine”: an app depends on an exact arrangement of files, and there was no way to hand that arrangement to someone else as one artifact.
- VMs solved it too, but at the wrong price — a full kernel and simulated hardware per copy, ~1 GB and ~30 seconds to run a 20 MB app.
- A container is just a Linux process started with namespaces (restricted view), cgroups (restricted budget), and a root filesystem built from image layers. There is no container object in the kernel.
- Four separate things are called “Docker”: the image format (now the open OCI standard), the build tool, the runtime, and the developer experience. Kubernetes dropped only the runtime piece in 2020; images were unaffected.
- The trade: a reproducible artifact and millisecond startup, in exchange for sharing the host kernel and accepting a stateless, externally-configured application shape.
Next: what a container actually is, in kernel terms — the namespaces, the cgroups, and the two-minute version you can reproduce by hand without Docker at all.