12. ACLs and security

A default Consul installation has no authentication and no encryption. Anyone who can reach port 8500 can read every key, register any service, and deregister everything. Anyone who can send packets to 8301 can join the gossip pool.

That is fine for an afternoon of experimentation and unacceptable for anything else. This chapter is the three layers that fix it.

   1. GOSSIP ENCRYPTION   a shared symmetric key on port 8301/8302
   2. TLS                 certificates for RPC (8300) and HTTP (8501)
   3. ACLs                tokens deciding what each caller may do

All three, or none of them is worth much. ACLs without TLS means tokens travel in plaintext. TLS without gossip encryption means anyone can join the cluster and be told everything. Gossip encryption alone protects membership and nothing else.

Gossip encryption

consul keygen
# 3Qc0dLxJ1LEnaTBRnrPVL/BsFBs4gDLXCaLDF9nq6/Y=

encrypt = "3Qc0dLxJ1LEnaTBRnrPVL/BsFBs4gDLXCaLDF9nq6/Y="
encrypt_verify_incoming = true
encrypt_verify_outgoing = true

Every agent needs the same key. Rotation is three ordered steps, and the order is not optional:

consul keyring -install=<new>     # 1. everyone can now DECRYPT with it
consul keyring -list              #    VERIFY every node has it
consul keyring -use=<new>         # 2. everyone now ENCRYPTS with it
consul keyring -remove=<old>      # 3. retire the old key

Running -use before every node has installed the key splits your cluster into two halves that cannot decrypt each other’s gossip. Always -list between steps.

TLS

tls {
  defaults {
    ca_file   = "/etc/consul.d/tls/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/tls/dc1-server-consul-0.pem"
    key_file  = "/etc/consul.d/tls/dc1-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  internal_rpc {
    verify_server_hostname = true
  }
}

ports {
  http  = -1        # disable plaintext HTTP entirely
  https = 8501
}

Consul generates its own CA if you want one:

consul tls ca create
consul tls cert create -server -dc dc1
consul tls cert create -client -dc dc1

verify_server_hostname = true is the one that matters most. Without it, any client certificate signed by your CA can impersonate a server — which means a compromised client agent can pretend to be a server and hand every other agent a forged catalog. It has been the subject of a real advisory, and it should be considered mandatory.

ports { http = -1 } disables the plaintext HTTP API. Leaving it on while enabling HTTPS gives you an unauthenticated plaintext path that people find by accident and then depend on.

The ACL system

Three objects, and the relationship between them is the whole model:

   POLICY   rules: what may be read/written, on which resources
   ROLE     a named bundle of policies
   TOKEN    a secret, carrying policies and/or roles

   token ──► roles ──► policies ──► rules
       └──────────────► policies

Turning it on

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"
  enable_token_persistence = true
}

consul acl bootstrap

AccessorID:   4d8e3f92-...
SecretID:     b7c9e1a4-...      ← the bootstrap token. Save it NOW.
Policies:     global-management

consul acl bootstrap can be run exactly once, and the SecretID is shown exactly once.

Lose it before creating another management token and you must perform a bootstrap reset: read the reset index from a server’s log, write it into <data_dir>/acl-bootstrap-reset, and bootstrap again — on every server. It is recoverable and it is an unpleasant hour. Put the token in a password manager or Vault before you do anything else.

down_policy = "extend-cache" is the setting that keeps you running during a partition. When an agent cannot reach the servers to validate a token, extend-cache keeps honouring cached results instead of denying everything. Without it, losing quorum turns into a total outage rather than a frozen catalog.

The migration nobody tells you about

Setting default_policy = "deny" on a running cluster stops everything at once. The order that works:

   1. acl { enabled = true, default_policy = "allow" }
      Nothing breaks. ACLs are evaluated but a miss is allowed.

   2. Bootstrap. Create policies and tokens for every component:
      each agent, each service, each operator, each tool.

   3. Distribute them. Watch the servers' logs for
      "ACL not found" and "Permission denied" warnings —
      these name exactly what you have missed.

   4. When the logs are quiet for a few days,
      switch to default_policy = "deny".

Step 3 is the one people skip, and it is the step that makes the difference between a clean switch and an outage. The logs during default_policy = "allow" are a complete list of everything that would break.

Writing policies

# an agent's own token
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}

# one service's token
service "orders" {
  policy = "write"          # register and deregister itself
}
service_prefix "" {
  policy = "read"           # discover others
}
node_prefix "" {
  policy = "read"
}
key_prefix "config/orders/" {
  policy = "read"
}
session_prefix "" {
  policy = "write"          # for locks and leader election
}

# read-only, for a dashboard
service_prefix "" { policy = "read" }
node_prefix ""    { policy = "read" }
key_prefix ""     { policy = "read" }

Resources: agent, node, service, key, session, event, query, acl, operator, mesh, peering, keyring. Each has an exact form and a _prefix form.

Levels: read, write (which includes read), deny, and list for key prefixes.

deny beats everything. A token whose policies grant write and also deny on the same resource is denied. That makes exceptions expressible:

key_prefix "config/" { policy = "write" }
key_prefix "config/secrets/" { policy = "deny" }

Roles

consul acl policy create -name svc-orders -rules @orders-policy.hcl
consul acl role create -name orders -policy-name svc-orders
consul acl token create -description "orders on web-1" -role-name orders

Use roles. A policy change then applies to every token carrying the role, with no token reissue. Attaching policies directly to tokens means a policy change requires finding and reissuing every token that referenced it.

Node identities and service identities

Shortcuts for the two most common policies, so you do not write them yourself:

consul acl token create -node-identity "web-1:dc1"
consul acl token create -service-identity "orders"

-service-identity "orders" generates the correct policy for a service to register itself, discover others, and participate in the mesh — including the sidecar’s permissions, which are easy to get wrong by hand.

Templated policies

service_prefix "" { policy = "read" }
service "${value}" { policy = "write" }

Combined with an auth method, this produces per-workload tokens automatically without a policy per service.

Auth methods

Tokens you create by hand must be distributed, rotated and revoked by you. Auth methods let a workload exchange an identity it already has for a Consul token, which removes the distribution problem entirely.

consul acl auth-method create \
  -name kubernetes -type kubernetes \
  -config '{
    "Host": "https://kubernetes.default.svc",
    "CACert": "...",
    "ServiceAccountJWT": "..."
  }'

consul acl binding-rule create \
  -method kubernetes \
  -bind-type service \
  -bind-name '${serviceaccount.name}'

Now a pod presents its Kubernetes ServiceAccount token and receives a Consul token scoped to a service named after that account. Nothing is distributed, nothing is long-lived, and revoking the ServiceAccount revokes Consul access.

Types: kubernetes, jwt, oidc, aws-iam, and Vault-issued. This is the right answer at any scale beyond a handful of services, and it is the same argument as chapter 7 of the Vault guide: a credential you never had to hand out is a credential you never have to rotate.

Using tokens

export CONSUL_HTTP_TOKEN=b7c9e1a4-...
consul kv get config/orders/timeout

curl -H "X-Consul-Token: b7c9e1a4-..." http://localhost:8501/v1/kv/config/orders/timeout

# in the agent's configuration
acl {
  tokens {
    agent          = "..."      # this agent's own operations
    default        = "..."      # anonymous requests to this agent
    config_file_service_registration = "..."
    dns            = "..."      # DNS queries specifically
  }
}

enable_token_persistence = true writes tokens set through the API into the data directory so they survive an agent restart — without it, a restarted agent has no token and cannot register anything.

The anonymous token

consul acl token update -id anonymous -policy-name dns-read

Requests without a token use the anonymous token. With default_policy = "deny" it grants nothing, which means DNS queries fail, because DNS carries no token.

That surprises people during the migration. Either give the anonymous token a narrow read policy, or set a dedicated tokens.dns on each agent — the second is better, because it does not grant anything to unauthenticated HTTP callers.

An audit checklist

   [ ] encrypt set, and rotated in three verified steps
   [ ] TLS on, verify_incoming and verify_outgoing
   [ ] verify_server_hostname = true          ← impersonation
   [ ] ports { http = -1 }                     ← no plaintext API
   [ ] acl.enabled, default_policy = "deny"
   [ ] down_policy = "extend-cache"
   [ ] bootstrap token in Vault or a password manager
   [ ] every service has its own token, not a shared one
   [ ] roles used, not policies attached directly to tokens
   [ ] auth methods rather than hand-distributed tokens
   [ ] anonymous token grants nothing, or a narrow DNS read
   [ ] enable_script_checks = false            ← RCE
   [ ] the UI is not on a public address
   [ ] 8500/8501/8300 not reachable from the internet
   [ ] mesh intentions default to deny
   [ ] snapshots encrypted at rest — they contain every token

Two entries deserve emphasis.

enable_script_checks = false. Enabling it lets anyone who can register a service through the API run commands as the agent’s user. Use enable_local_script_checks if you need script checks at all; it accepts them only from local configuration files.

Snapshots contain every ACL token. A snapshot in an unencrypted bucket is a complete credential dump for your cluster. Encrypt them, and restrict who can read them the way you would restrict the tokens themselves.

What to take from this chapter

Three layers: gossip encryption, TLS, ACLs. Any one alone leaves a gap the others were covering.
Rotate the gossip key in three verified steps. -use before every node has -installed splits the cluster.
verify_server_hostname = true, or a client certificate can impersonate a server.
consul acl bootstrap runs once and shows the secret once. Store it before doing anything else.
Migrate with default_policy = "allow" first and use the “Permission denied” log lines as your checklist.
down_policy = "extend-cache" turns a partition into degraded operation rather than a total outage.
deny beats write, which makes exceptions expressible.
Use roles, and use auth methods — a token you never distributed is a token you never rotate.
The anonymous token is what DNS uses. Deny-by-default breaks DNS until you handle it.
Snapshots are a full credential dump. Encrypt them.

Next: tuning and operations — sizing, upgrades, backups and the metrics that tell you a cluster is unhealthy before it is.

Written by Sepuri Sai Krishna