10. The service mesh

Discovery tells a service where to connect. The mesh decides whether it is allowed to, encrypts the connection, and gives you the traffic controls in chapter 11.

It is also the largest commitment in Consul, and this chapter tries to be honest about that.

The problem it solves

   WITHOUT A MESH

   orders ──plaintext──► inventory
      │
      ├── anything on the network can read it
      ├── anything on the network can CALL inventory
      ├── inventory has no idea who is calling
      └── the firewall rule is by IP, and IPs change

   the usual answers:
     TLS in every application    → every team implements it differently,
                                   certificate rotation is nobody's job
     network policy by IP        → IPs change; policy lags reality
     an API gateway              → only helps at the edge, not between
                                   internal services

What Consul does about it

   ┌── pod / VM ────────┐          ┌── pod / VM ────────┐
   │  orders            │          │  inventory         │
   │    │ localhost     │          │        ▲ localhost │
   │    ▼               │          │        │           │
   │  ENVOY sidecar ────┼──mTLS───►┼─── ENVOY sidecar   │
   └────────────────────┘          └────────────────────┘
          ▲                                  ▲
          └────────── Consul servers ────────┘
             certificates, endpoints, intentions

Every service gets an Envoy sidecar. The application speaks plain HTTP to localhost; the sidecar handles TLS, identity, authorisation, retries and load balancing. Consul configures the sidecars over xDS — the same protocol described in chapter 3 of the Envoy guide, with Consul as the control plane.

Consul does not implement the data plane. It configures Envoy. Everything Envoy can do, it can do here; everything Envoy cannot do, neither can Consul’s mesh.

Identity, and where it comes from

Every service gets an X.509 certificate whose SAN encodes its identity as a SPIFFE URI:

spiffe://a1b2c3d4-....consul/ns/default/dc/dc1/svc/orders
                                                    └── the service name

The certificate is the identity. There is no shared secret, no API key, and nothing to leak that is valid for more than a few hours.

consul connect ca get-config

{
  "Provider": "consul",
  "Config": {
    "LeafCertTTL": "72h",
    "IntermediateCertTTL": "8760h",
    "RotationPeriod": "2160h"
  }
}

Four CA providers:

   consul     built in. Zero configuration. Good enough for most.
   vault      Vault's PKI engine is the CA. The right answer if you
              already run Vault — one CA, one audit trail, real HSM
              support. See the Vault guide's PKI chapter.
   aws-pca    AWS Private CA. Expensive; use it if compliance requires.
   provider   an external CA via the API

Leaf certificates are rotated automatically, well before their 72-hour lifetime. Nobody runs a renewal cron job and nobody is ever paged for an expired internal certificate — which, if you have ever been paged for one, is a substantial argument on its own.

The Vault provider is worth considering seriously if Vault is already in your stack: chapter 8 of the Vault guide argues that short-lived certificates make revocation mostly unnecessary, and Consul’s mesh is that argument implemented.

Intentions

An intention is a rule about which service may call which. Not which IP — which service.

consul intention create -allow web orders
consul intention create -deny  web payments
consul intention create -allow '*' logging
consul intention check web orders
consul intention list

Or as a configuration entry, which is the version you should keep in version control:

Kind = "service-intentions"
Name = "orders"

Sources = [
  { Name = "web",       Action = "allow" },
  { Name = "admin-api", Action = "allow" },
  { Name = "*",         Action = "deny"  },
]

Enforcement is in the callee’s sidecar, which is the correct place: the caller cannot bypass it by lying, because its identity comes from a certificate it cannot forge.

Default deny

Kind = "service-intentions"
Name = "*"
Sources = [{ Name = "*", Action = "deny" }]

consul config write default-deny.hcl

This is the setting that makes the mesh worth having, and it is not the default because turning it on in an existing system breaks everything at once.

The migration that works:

   1. Mesh everything. Leave intentions default-allow.
   2. Run for weeks. Collect who actually calls whom, from
      the sidecars' metrics and access logs.
   3. Write explicit allow intentions for every real path.
   4. Verify: consul intention check <src> <dst> for each.
   5. THEN switch the wildcard to deny.

Skipping step 2 means writing intentions from an architecture diagram, and the architecture diagram is always wrong — there is always one service calling another that nobody documented.

L7 intentions

With Protocol = "http" declared, intentions can be per-path and per-method:

Kind = "service-intentions"
Name = "orders"

Sources = [
  {
    Name = "web"
    Permissions = [
      { Action = "allow", HTTP = { PathPrefix = "/api/v1/orders", Methods = ["GET"] } },
      { Action = "allow", HTTP = { PathExact = "/api/v1/orders", Methods = ["POST"] } },
    ]
  },
  {
    Name = "reporting"
    Permissions = [
      { Action = "allow", HTTP = { PathPrefix = "/api/v1/orders", Methods = ["GET"] } },
    ]
  },
]

Reporting may read orders and may not create them, enforced at the network layer rather than in application code. This is genuinely valuable and it is also configuration that drifts — an endpoint added without a matching intention fails in production with a 403 that looks like an application bug.

Registering a service in the mesh

service {
  name = "orders"
  port = 8080

  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "inventory"
            local_bind_port  = 9001
          },
          {
            destination_name = "payments"
            local_bind_port  = 9002
          }
        ]
      }
    }
  }
}

consul connect envoy -sidecar-for orders

The application now calls localhost:9001 to reach inventory. No hostname, no discovery, no TLS in application code. The sidecar resolves, encrypts, authorises, load balances and retries.

That explicit upstream list is also the friction: every new dependency is a configuration change and a restart.

Transparent proxy

Which is why transparent proxy exists.

connect {
  sidecar_service {
    proxy {
      mode = "transparent"
    }
  }
}

   iptables rules redirect ALL outbound traffic into the sidecar
        │
   the sidecar resolves `inventory.virtual.consul` itself
        │
   → no upstream list, no local_bind_port, no restart to add
     a dependency

The application calls http://inventory — or in Kubernetes, the ordinary Service name — and it just works.

This is the mode to use in Kubernetes, where it is the default in recent chart versions. The cost is that traffic interception is now iptables magic, and when something does not work the debugging is harder: a connection that should be direct is being intercepted, or one that should be intercepted is not.

metadata:
  annotations:
    consul.hashicorp.com/connect-inject: "true"
    consul.hashicorp.com/transparent-proxy: "true"
    consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: "9090"

The exclusion annotations matter more than they look — a metrics scrape, a database on a non-mesh host, or a call to an external API often needs to bypass the sidecar.

Gateways

Three kinds, for three boundary problems:

Ingress gateway — outside traffic into the mesh:

Kind = "ingress-gateway"
Name = "public"

Listeners = [{
  Port     = 8443
  Protocol = "http"
  Services = [
    { Name = "web",     Hosts = ["www.example.com"] },
    { Name = "api",     Hosts = ["api.example.com"] },
  ]
}]

Terminating gateway — mesh services to things outside it, so a legacy database gets mesh identity and intentions applied to it even though it has no sidecar:

Kind = "terminating-gateway"
Name = "legacy"
Services = [{ Name = "mainframe-db", CAFile = "/etc/ca.pem" }]

Mesh gateway — between datacentres, so services in dc1 reach dc2 without every pod needing routable connectivity:

Kind = "mesh"
Peering = { PeerThroughMeshGateways = true }

Mesh gateways are the piece that makes multi-datacentre practical. Without them, cross-datacentre mesh traffic requires full pod-to-pod routing between regions. With them, one address per datacentre is enough, and the traffic stays encrypted end to end because the gateway routes by SNI without decrypting — the same trick as chapter 9 of the HAProxy guide.

Cluster peering

The modern alternative to WAN federation for connecting datacentres:

# in dc1
consul peering generate-token -name dc2 > token.txt

# in dc2
consul peering establish -name dc1 -peering-token "$(cat token.txt)"

   WAN FEDERATION              CLUSTER PEERING

   one primary datacentre      no primary; equals
   shared ACL and CA state     independent, each its own
   tight coupling              loose coupling
   all-or-nothing              export only chosen services

Kind = "exported-services"
Name = "default"
Services = [{
  Name      = "orders"
  Consumers = [{ Peer = "dc2" }]
}]

Peering is the better default for new multi-cluster deployments. Only the services you export are visible, the clusters have no shared state to get out of sync, and losing one has no effect on the other’s ability to operate independently.

What adopting the mesh actually costs

The honest list, because the benefits are always the part that gets presented:

A proxy in every request path. Roughly 1–3 ms added latency and 50–100 MB of memory per sidecar. At a thousand pods that is 50–100 GB of memory doing nothing but proxying.

Envoy becomes something you must understand. When a request fails, the answer is in Envoy’s stats and access logs, not in your application’s. Chapter 11 of the Envoy guide becomes required reading, and the whole Envoy guide becomes useful background.

Debugging is genuinely harder. A 503 might be your service, the local sidecar, the remote sidecar, an intention, a certificate, or a configuration entry. There are more places for it to be wrong.

Certificate and configuration state must be healthy. A CA rotation that goes wrong takes down every connection in the mesh at once, because every connection depends on it.

It is a permanent operational commitment. Sidecar upgrades, Envoy version compatibility, and a new failure surface, forever.

What you get for that: mTLS everywhere with automatic rotation, service-identity authorisation that survives IPs changing, per-service traffic control, and uniform telemetry for every call in the system without instrumenting anything.

Whether that trade is worth it depends on how many services you have and what your compliance requirements are, and chapter 15 gives a straight answer.

What to take from this chapter

• Consul configures Envoy; it does not implement a data plane. Envoy’s capabilities and limits are the mesh’s.
• Identity is an X.509 certificate with a SPIFFE URI, rotated automatically well before its 72-hour life. Nobody is ever paged for an expired internal certificate.
• Use the Vault CA provider if you already run Vault — one CA, one audit trail.
• Intentions authorise by service identity, enforced in the callee’s sidecar, which the caller cannot bypass.
• Default-deny is the point of the mesh, and the migration must go through weeks of observation first — the architecture diagram is always wrong.
• Transparent proxy removes the explicit upstream list and is the right mode in Kubernetes, at the cost of iptables-shaped debugging.
• Mesh gateways make multi-datacentre practical; cluster peering is the better default over WAN federation for new deployments.
• The cost is a proxy in every path, Envoy expertise, harder debugging and a permanent commitment. Count it before adopting.

Next: traffic management — resolvers, splitters and routers, and the canary deploy they add up to.