9. The key-value store

Consul’s KV store is small, strongly consistent, and built for coordination rather than data. It is the third product in the binary and the one most people underuse.

The basics

consul kv put config/orders/timeout 30
consul kv get config/orders/timeout
consul kv get -detailed config/orders/timeout
consul kv get -recurse config/orders/
consul kv delete config/orders/timeout
consul kv delete -recurse config/orders/
consul kv export config/ > backup.json
consul kv import @backup.json

curl -X PUT http://localhost:8500/v1/kv/config/orders/timeout -d '30'
curl http://localhost:8500/v1/kv/config/orders/timeout | jq -r '.[0].Value' | base64 -d

Values come back base64-encoded through the HTTP API. That is not a quirk to work around — it is how binary values are carried safely — but it catches everyone once.

There are no directories. The / in a key is a convention, and ?recurse does a prefix match. config/orders/ and config/orders are different prefixes and the trailing slash matters.

The limits

   value size          512 KB hard limit
   practical value     a few KB
   total store         hundreds of MB, all in memory on every server
   write throughput    every write is a Raft commit — hundreds/sec
   read throughput     very high (stale reads scale across servers)

Every write goes through Raft and every value sits in memory on every server. This is a coordination store, not a database. If you are considering putting a megabyte of anything in it, or writing to it thousands of times a second, it is the wrong tool — and chapter 15 says what the right one is.

Check-and-set

The primitive everything else is built on.

consul kv get -detailed config/orders/timeout
# ModifyIndex  128401

consul kv put -cas -modify-index=128401 config/orders/timeout 60
# Success! ... true

Every key carries a ModifyIndex. A CAS write succeeds only if the current index matches what you supply. This is optimistic concurrency, and it turns read-modify-write from a race into a retry loop:

while True:
    kv = get("config/orders/limits")
    limits = json.loads(kv["Value"])
    limits["max"] += 10
    if put_cas("config/orders/limits", json.dumps(limits), kv["ModifyIndex"]):
        break
    # somebody else wrote first. Read again and retry.

-cas -modify-index=0 means “create only if absent”, which is the atomic-create primitive.

Transactions

Several operations, all or nothing:

curl -X PUT http://localhost:8500/v1/txn -d '[
  {"KV": {"Verb": "cas", "Key": "config/a", "Index": 128401, "Value": "MQ=="}},
  {"KV": {"Verb": "set", "Key": "config/b", "Value": "Mg=="}},
  {"KV": {"Verb": "delete", "Key": "config/c"}}
]'

Up to 64 operations. Verbs include set, cas, lock, unlock, get, check-index, check-session, delete, delete-cas. Transactions can also touch the catalog and checks, which makes “register this service and set its configuration atomically” a single operation.

Blocking queries on the KV store

The same mechanism as chapter 7, and this is what makes the KV store useful as a configuration system.

curl -i "http://localhost:8500/v1/kv/config/orders/timeout"
# X-Consul-Index: 128401

curl "http://localhost:8500/v1/kv/config/orders/timeout?index=128401&wait=5m"
# returns the moment the value changes

# consul-template
{{ key "config/orders/timeout" }}
{{ with $t := keyOrDefault "config/orders/timeout" "30" }}timeout = {{ $t }}{{ end }}
{{ range tree "config/orders" }}{{ .Key }} = {{ .Value }}
{{ end }}

Change a value, and every application holding a watch has the new configuration within milliseconds — no deploy, no restart, no poll. That is a genuinely good configuration-management story and it is the reason to use the KV store at all rather than a file.

?recurse&index= watches a whole prefix, so one watcher covers a service’s entire configuration tree.

Sessions

A session is a lease with a liveness condition, and it is what makes locking possible.

curl -X PUT http://localhost:8500/v1/session/create -d '{
  "Name": "orders-leader",
  "Node": "web-1",
  "Checks": ["serfHealth", "orders-ready"],
  "LockDelay": "15s",
  "TTL": "30s",
  "Behavior": "release"
}'
# {"ID": "adf4238a-882b-9ddc-4a9d-5b6758e4159e"}

Four fields decide the semantics:

Checks — the session is invalidated the moment any listed check goes critical. serfHealth is included by default, so a session dies when its node stops gossiping. Adding your service’s own check is the useful part: a leader whose service is failing loses its leadership automatically, without anyone noticing and intervening.

TTL — the session must be renewed within this window or it is invalidated. Belt as well as braces.

Behavior — release (the default) frees held locks when the session dies; delete deletes the keys outright, which is how you build ephemeral registration.

LockDelay — the interesting one, below.

curl -X PUT http://localhost:8500/v1/session/renew/<id>
curl -X PUT http://localhost:8500/v1/session/destroy/<id>
curl http://localhost:8500/v1/session/info/<id>

Locks and leader election

# acquire: succeeds only if the key is not already held
curl -X PUT "http://localhost:8500/v1/kv/service/orders/leader?acquire=<session>" \
  -d '{"node": "web-1", "pid": 4412}'
# true  → you hold it
# false → somebody else does

curl -X PUT "http://localhost:8500/v1/kv/service/orders/leader?release=<session>"

The whole election, as a loop:

session = create_session(checks=["serfHealth", "orders-ready"], ttl="30s")

while True:
    if acquire("service/orders/leader", session, my_identity()):
        become_leader()
        while renew(session):        # keep renewing while leading
            do_leader_work()
        step_down()                  # renewal failed — we are not leader
    else:
        # BLOCKING query: wake the instant the current leader lets go
        watch("service/orders/leader", index=current_index)

The blocking query in the else branch is what makes failover fast. The moment the leader’s session is invalidated, the key is released, its index changes, and every waiting candidate is woken in milliseconds rather than at the next poll.

The command-line versions handle all of this:

consul lock service/orders/leader ./my-leader-process
consul lock -n 3 service/workers/slots ./worker    # a semaphore: 3 holders

consul lock acquires, runs your command while holding, and sends it a signal when the lock is lost. It is the right way to make an existing program singleton-safe without writing any Consul code.

LockDelay

   the leader's node dies
        │
   session invalidated, lock released
        │
   ┌──── LockDelay (15s by default) ────┐
   │  NOBODY may acquire this lock       │
   └─────────────────────────────────────┘
        │
   a new leader may now acquire

Why deliberately wait? Because a session can be invalidated by a network partition rather than by the leader actually stopping. The old leader may still be running and may not yet know it has lost the lock. LockDelay gives it time to notice.

It reduces the window. It does not close it, and no amount of delay can, because you cannot bound how long a process may be paused by GC, a stalled disk or a hypervisor.

The fencing token

This is the part almost every distributed-lock tutorial omits, and it is what separates a lock that works from one that mostly works.

   process A holds the lock
        │
   A is paused (GC, hypervisor, stalled disk) for 60 seconds
        │
   A's session TTL expires. Lock released.
   LockDelay passes. Process B acquires it.
        │
   B starts doing the work
        │
   A WAKES UP. It still believes it holds the lock.
   It writes.
        │
   → two writers. The lock did not prevent it.

A distributed lock cannot stop a paused process from acting when it wakes. The only fix is to make the resource reject stale writes.

Consul’s KV store provides the token for free: ModifyIndex on the lock key only ever increases.

result = acquire("service/orders/leader", session)
kv = get("service/orders/leader")
fencing_token = kv["ModifyIndex"]     # monotonically increasing

# every write to the protected resource carries it
db.execute(
    "UPDATE resource SET data = %s, fence = %s WHERE fence < %s",
    (data, fencing_token, fencing_token)
)

   A acquires   → token 128401
   A pauses
   B acquires   → token 128455        (always higher)
   B writes with 128455 → accepted, fence is now 128455
   A wakes, writes with 128401 → REJECTED, 128401 < 128455

If the resource does not check a fencing token, you do not have a lock — you have a strong suggestion. That is true of Consul, of etcd, of ZooKeeper and of Redis, and it is the single most important sentence about distributed locking. Chapter 8 of the etcd guide makes the same argument with etcd’s revision numbers.

For many uses a strong suggestion is genuinely enough — an occasional duplicate cache refresh harms nobody. Decide which case you are in deliberately, rather than assuming the lock is absolute.

What to put in it

Good:

   feature flags                config/features/new-checkout = true
   tuning values                config/orders/rate-limit = 1000
   leader election keys         service/orders/leader
   distributed locks            locks/nightly-reconciliation
   small service configuration  config/orders/*
   maintenance flags            ops/maintenance/orders = true

Bad:

   anything over a few KB       it is in memory on every server
   high-frequency writes        every write is a Raft commit
   secrets                      → Vault, chapter 15
   application data             → a database
   large blobs                  → object storage
   session data                 → Redis

Secrets in particular. The Consul KV store has ACLs but no encryption at rest by default, no audit log in the open-source version, no automatic rotation, and no dynamic credentials. Vault exists for exactly this, is from the same vendor, and integrates directly.

Watching a key change things

The pattern worth adopting:

# consul-template
template {
  source      = "/etc/app/config.json.ctmpl"
  destination = "/etc/app/config.json"
  command     = "app-validate-config /etc/app/config.json && systemctl reload app"
}

{
  "timeout": {{ keyOrDefault "config/orders/timeout" "30" }},
  "rate_limit": {{ keyOrDefault "config/orders/rate-limit" "1000" }},
  "features": {
    {{ range $i, $p := tree "config/features" }}{{ if $i }},{{ end }}
    "{{ $p.Key }}": {{ $p.Value }}{{ end }}
  }
}

keyOrDefault, not key. A missing key with key renders an empty value into your configuration file, and the reload then fails or, worse, succeeds with nonsense. Always supply a default, and always validate before reloading.

What to take from this chapter

• It is a coordination store, not a database. 512 KB per value, everything in memory on every server, every write a Raft commit.
• Check-and-set with ModifyIndex turns read-modify-write into a retry loop. -modify-index=0 is atomic create.
• Blocking queries on keys are the point — change a value and every watcher has it in milliseconds.
• Sessions bind a lock to health checks, so a leader whose service fails loses leadership automatically.
• LockDelay narrows the window after an invalidation; it cannot close it.
• A lock without a fencing token the resource checks is a strong suggestion, not a lock. ModifyIndex is the token, and it costs one column in your table.
• consul lock <key> <command> makes an existing program singleton-safe with no code.
• Secrets belong in Vault, not here. Use keyOrDefault and validate before reloading.

Next: the service mesh — sidecar proxies, mTLS from a built-in CA, and intentions as a deny-by-default authorisation model.