1. What Consul is, and why it exists

Consul answers a question that sounds trivial until you have more than three machines.

Where is the thing I need to call, and is it working right now?

Every distributed system needs that answer, and for most of computing history the answer was a configuration file somebody edited by hand.

The problem

You have a web service that calls an orders service. So you write down where the orders service is.

orders_service_urls:
  - http://10.0.1.11:8080
  - http://10.0.1.12:8080
  - http://10.0.1.13:8080

This works. It keeps working until any of the following happens, and all of them happen:

   a machine dies              → the list still contains it,
                                 one request in three fails

   you scale up                → the new machine gets no traffic
                                 until somebody edits and redeploys

   an instance is unhealthy    → it accepts TCP, returns 500s,
     but not dead                and stays in the list

   IPs change on reboot        → the list is now fiction

   you add a datacentre        → the file must know which
                                 instances are near

Every one of those is a person editing a file and redeploying, which means minutes to hours during which the system is wrong. The list is a cache of reality that nobody invalidates.

Load balancers solve some of this, and they move the problem: now something must keep the load balancer’s backend list current, and that something is a person editing a file.

The answer

Turn the list into a query.

   BEFORE                          AFTER

   read a config file              ask a service registry
   containing three IPs            "who is healthy and named orders?"

   updated by a human              updated by the services themselves
   correct when written            correct right now

Consul is that registry, and the part it gets right that a simple registry does not is the second half of the question. Registration is easy. Knowing that a registered thing is still working is the hard part, and Consul was built around continuous health checking from the first release.

   orders-1  registers  ──► ┌──────────────┐
   orders-2  registers  ──► │    CONSUL    │  ◄── "give me healthy
   orders-3  registers  ──► │   catalog    │       orders instances"
                            └──────┬───────┘
                                   │
                    health checks every 10s
                    orders-2 fails → removed from answers
                    within seconds, with no human involved

Where it came from

HashiCorp released Consul in 2014, built on Serf, their gossip library from the year before. That lineage explains the architecture: rather than every node heartbeating to a central server, Consul nodes gossip with each other, so failure detection is distributed and scales to thousands of nodes without the servers becoming a bottleneck. Chapter 3 is about that mechanism, which is genuinely clever and is the thing Consul does differently from etcd.

It has accumulated three more products since:

   2014   service discovery + health checking + KV store
   2018   Consul Connect — service mesh, mTLS between services
   2020   Envoy as the standard sidecar; L7 traffic management
   2021   transparent proxy, admin partitions
   2023   licence changes from MPL 2.0 to BUSL 1.1

The licence matters and this guide will not skip it. Consul 1.17 and later are under the Business Source License, which is not open source: you may not use it to provide a competing service, and the terms convert to MPL after four years. For most organisations running Consul internally this changes nothing. For anyone building a product on top of it, or with a policy against non-OSI licences, it is decisive. The community fork is OpenBao for Vault; Consul’s situation is less settled, and the practical alternatives are named in chapter 15.

The four products

Consul is one binary doing four jobs, and you can use any subset.

1. Service discovery. A catalog of what exists and where, queryable by DNS or HTTP.

2. Health checking. Continuous, distributed, several protocols, and — the part that makes it different — checks run on the local agent, not from a central poller.

3. A key-value store. Small, strongly consistent, with blocking queries and sessions. Used for configuration, feature flags and leader election.

4. A service mesh. Sidecar Envoy proxies, mTLS with a built-in certificate authority, and a deny-by-default authorisation model called intentions.

Most installations start at 1 and 2, add 3, and adopt 4 much later if at all. That is the right order, and chapter 15 is direct about the fact that the mesh is a large commitment that many people do not need.

The one architectural idea

Almost every Consul question resolves through this diagram:

   ┌─ every machine runs an AGENT ───────────────────────────┐
   │                                                          │
   │   app ──► localhost:8500 ──► agent                       │
   │                                │                         │
   │   the agent:                   │                         │
   │     runs the health checks for services on THIS machine  │
   │     answers queries from a local cache                   │
   │     gossips with every other agent                       │
   └────────────────────────────────┬─────────────────────────┘
                                    │
                       ┌────────────▼────────────┐
                       │   SERVERS (3 or 5)      │
                       │   Raft, one leader      │
                       │   hold the catalog + KV │
                       └─────────────────────────┘

Applications always talk to localhost. They never need to know where the servers are, they never do service discovery to find the discovery service, and a request never crosses the network to be answered. That single decision is why Consul feels different to use from etcd, where clients connect to the cluster directly.

Health checks run locally too. The agent on the machine running orders-2 is the one checking orders-2, and it reports the result. Compare that with a central poller checking ten thousand endpoints from one place: the local agent scales linearly with your fleet, sees the service the way a neighbour would, and keeps working during a network partition — it just cannot report.

A first configuration

A server:

/etc/consul.d/server.hcl

datacenter = "dc1"
data_dir   = "/opt/consul"
server     = true
bootstrap_expect = 3
bind_addr  = "10.0.1.5"
client_addr = "0.0.0.0"
ui_config { enabled = true }
retry_join = ["10.0.1.5", "10.0.1.6", "10.0.1.7"]

A client agent, on every other machine:

/etc/consul.d/client.hcl

datacenter = "dc1"
data_dir   = "/opt/consul"
retry_join = ["10.0.1.5", "10.0.1.6", "10.0.1.7"]

A service, registered by dropping a file next to the agent:

/etc/consul.d/orders.hcl

service {
  name = "orders"
  port = 8080
  tags = ["v2", "primary"]

  check {
    id       = "orders-http"
    http     = "http://localhost:8080/healthz"
    interval = "10s"
    timeout  = "2s"
  }
}

And now the list is a query:

# DNS — no client library, no code change
dig @127.0.0.1 -p 8600 orders.service.consul

# only healthy instances are returned. That is the default.
curl http://localhost:8500/v1/health/service/orders?passing

The DNS interface is the feature that gets Consul adopted. Any language, any framework, any legacy application that can resolve a hostname can do service discovery with no code change at all — and chapter 7 is honest about the cases where that convenience quietly betrays you.

What it is not

It is not a load balancer. It tells you the healthy addresses; something else must distribute across them. Your client library does it, or HAProxy does it from a template, or the mesh’s Envoy sidecar does it.

It is not a general database. The KV store holds small values — the practical limit is 512 KB per key and you should be nowhere near it. It is for configuration and coordination, not data.

It is not a monitoring system. Its health checks decide routing, not alerting. They answer “should this receive traffic?” and that is a narrower question than “is this healthy?” — a distinction chapter 6 takes seriously.

It is not lightweight to operate. Three or five servers, an agent on every machine, gossip encryption keys, TLS certificates, an ACL system. That is real work, and it is the honest counterweight to everything above.

What to take from this chapter

• Service discovery replaces a hand-edited list with a query, and the hard half is not registration but knowing what still works.
• Health checks run on the local agent, which scales with the fleet and sees services the way a neighbour does.
• Applications talk to localhost and never to the servers. No bootstrapping problem, no network hop to get an answer.
• Four products in one binary: discovery, health checking, KV, and a mesh. Adopt them in that order.
• The DNS interface is why it gets adopted — service discovery with no code change in any language.
• The licence is BUSL since 1.17, which is not open source and matters decisively to some organisations.
• It is not a load balancer, a database, or a monitoring system, and running it is real operational work.

Next: the architecture — agents and servers, what each holds, and why the split is the whole design.