11. Traffic management

With the mesh in place and a protocol declared, Consul can control which instances receive a request. Three configuration entries do it, and they compose in a fixed order.

   a request from a sidecar
        │
        ▼
   ROUTER      match on path, header, method → choose a service subset
        │      (requires Protocol = "http")
        ▼
   SPLITTER    divide traffic by weight across subsets
        │
        ▼
   RESOLVER    define what the subsets ARE, plus failover and
        │      redirection
        ▼
   actual instances

Router, then splitter, then resolver, always. Remembering the order explains why a splitter with no resolver defining its subsets does nothing, and why a router referencing a split that does not exist fails silently.

Prerequisite: declare the protocol

Kind     = "service-defaults"
Name     = "orders"
Protocol = "http"

Without this, Consul treats the service as opaque TCP. Splitters and resolvers still work at the connection level; routers do nothing at all, because there is no path or header to match on. This is the most common reason traffic management “does not work”.

service-resolver

Defines subsets — named groups of instances, selected by a filter over their registration.

Kind          = "service-resolver"
Name          = "orders"
DefaultSubset = "v1"

Subsets = {
  v1 = { Filter = "Service.Meta.version == \"1.4.2\"" }
  v2 = { Filter = "Service.Meta.version == \"1.5.0\"" }
  canary = { Filter = "\"canary\" in Service.Tags" }
  eu = { Filter = "Node.Meta.region == \"eu-west\"" }
}

ConnectTimeout = "5s"

The filter language is the same one from chapter 5 — Service.Meta.*, Service.Tags, Node.Meta.*, Service.Port.

Subsets come from registration metadata, which means the deploy sets them. A new version registers with version = "1.5.0" and is automatically in the v2 subset with no Consul change at all. That is the property that makes this usable in a pipeline.

Failover

Kind = "service-resolver"
Name = "orders"

Failover = {
  "*" = {
    Datacenters = ["dc2", "dc3"]
  }
  "v2" = {
    Targets = [
      { Service = "orders", ServiceSubset = "v1" },
      { Datacenter = "dc2" },
    ]
  }
}

No healthy instances in this subset → try these, in order. Cross-datacentre, cross-subset, or cross-service. The sidecar handles it, so the application sees a successful request that happened to be served elsewhere.

v2 failing over to v1 is the pattern worth noticing: a bad canary that has no healthy instances falls back to the known-good version automatically, rather than returning errors while somebody notices.

Redirect

Kind = "service-resolver"
Name = "orders"
Redirect = { Service = "orders-v2" }

Everything asking for orders gets orders-v2 instead. Blunt, and the correct tool for a rename or a migration where you want one switch rather than a gradual shift.

service-splitter

Divides traffic by weight.

Kind = "service-splitter"
Name = "orders"

Splits = [
  { Weight = 95, ServiceSubset = "v1" },
  { Weight = 5,  ServiceSubset = "v2" },
]

Weights must sum to 100 and support one decimal place, so 0.1 is the smallest meaningful slice — one request in a thousand, which is a genuinely useful first canary step.

A splitter can also add headers, which is how you make a canary visible downstream:

Splits = [
  {
    Weight = 95
    ServiceSubset = "v1"
    ResponseHeaders = { Set = { "x-version" = "v1" } }
  },
  {
    Weight = 5
    ServiceSubset = "v2"
    ResponseHeaders = { Set = { "x-version" = "v2" } }
    RequestHeaders  = { Set = { "x-canary" = "true" } }
  },
]

Setting a request header on the canary path is worth doing, because it propagates: downstream services and their telemetry can distinguish canary traffic, and your dashboards can split error rates by it. Without it, a canary’s errors are 5% of a blended number and much harder to see.

service-router

Matches on request attributes. HTTP only.

Kind = "service-router"
Name = "orders"

Routes = [
  # internal testers get v2 explicitly
  {
    Match = {
      HTTP = { Header = [{ Name = "x-beta-user", Exact = "true" }] }
    }
    Destination = { Service = "orders", ServiceSubset = "v2" }
  },

  # the admin path goes to a different service entirely
  {
    Match = { HTTP = { PathPrefix = "/admin" } }
    Destination = { Service = "orders-admin" }
  },

  # slow reports get a longer timeout and no retries
  {
    Match = { HTTP = { PathPrefix = "/reports" } }
    Destination = {
      Service        = "orders"
      RequestTimeout = "120s"
      NumRetries     = 0
    }
  },

  # everything else: retry on connection failure and 503
  {
    Match = { HTTP = { PathPrefix = "/" } }
    Destination = {
      Service           = "orders"
      NumRetries        = 3
      RetryOnConnectFailure = true
      RetryOnStatusCodes    = [503]
    }
  },
]

Match criteria: PathExact, PathPrefix, PathRegex, Header (Exact, Prefix, Suffix, Regex, Present, Invert), QueryParam, Methods.

Destination options: Service, ServiceSubset, Namespace, PrefixRewrite, RequestTimeout, IdleTimeout, NumRetries, RetryOnConnectFailure, RetryOnStatusCodes, RetryOn.

Routes are evaluated in order, first match wins, and an unmatched request goes to the default. Put the specific routes first and a catch-all last, exactly as in every routing system.

Be careful with retries. RetryOnStatusCodes = [503] on a non-idempotent endpoint means a POST that timed out after being processed gets sent again. Retry connection failures freely; retry status codes only where re-execution is safe — the same argument chapter 7 of the HAProxy guide and chapter 8 of the Airflow guide both make.

A canary deploy, complete

The three entries together:

# 1. subsets, defined by what the deploy registers
Kind = "service-resolver"
Name = "orders"
DefaultSubset = "stable"
Subsets = {
  stable = { Filter = "Service.Meta.version == \"1.4.2\"" }
  canary = { Filter = "Service.Meta.version == \"1.5.0\"" }
}
Failover = {
  "canary" = { Targets = [{ Service = "orders", ServiceSubset = "stable" }] }
}

# 2. the split, which is the thing you edit during the rollout
Kind = "service-splitter"
Name = "orders"
Splits = [
  { Weight = 99, ServiceSubset = "stable" },
  { Weight = 1,  ServiceSubset = "canary",
    RequestHeaders = { Set = { "x-canary" = "true" } } },
]

# 3. internal testers can force the canary regardless of the split
Kind = "service-router"
Name = "orders"
Routes = [
  {
    Match = { HTTP = { Header = [{ Name = "x-force-canary", Exact = "true" }] } }
    Destination = { Service = "orders", ServiceSubset = "canary" }
  },
]

The rollout is then one file, edited:

for w in 1 5 25 50 100; do
  sed -i "s/Weight = [0-9]*, ServiceSubset = \"canary\"/Weight = $w, ServiceSubset = \"canary\"/" splitter.hcl
  sed -i "s/Weight = [0-9]*, ServiceSubset = \"stable\"/Weight = $((100-w)), ServiceSubset = \"stable\"/" splitter.hcl
  consul config write splitter.hcl
  sleep 300
  ./check-error-rate.sh || { git checkout splitter.hcl; consul config write splitter.hcl; exit 1; }
done

Rolling back is writing the old splitter, which takes effect in the sidecars within a second or two. That is faster than any deploy-based rollback and it is the strongest practical argument for traffic management at all.

Other useful entries

Circuit breaking and connection limits:

Kind = "service-defaults"
Name = "orders"
Protocol = "http"

UpstreamConfig = {
  Defaults = {
    Limits = {
      MaxConnections        = 100
      MaxPendingRequests    = 100
      MaxConcurrentRequests = 100
    }
    PassiveHealthCheck = {
      Interval    = "10s"
      MaxFailures = 5
    }
  }
}

PassiveHealthCheck ejects an instance from the sidecar’s local pool after consecutive failures — Envoy’s outlier detection, which is a second layer of protection independent of Consul’s health checks and reacts faster because it observes real traffic. Chapter 9 of the Envoy guide covers the mechanism.

Sticky sessions:

Kind = "service-resolver"
Name = "orders"
LoadBalancer = {
  Policy = "maglev"
  HashPolicies = [{
    Field      = "header"
    FieldValue = "x-session-id"
  }]
}

maglev and ring_hash are consistent-hashing policies, so adding or removing an instance moves only that instance’s share — the same requirement as hash-type consistent in chapter 6 of the HAProxy guide. Other policies: round_robin, least_request, random.

Mesh-wide TLS floor:

Kind = "mesh"
TLS = {
  Incoming = { TLSMinVersion = "TLSv1_2" }
  Outgoing = { TLSMinVersion = "TLSv1_2" }
}
HTTP = { SanitizeXForwardedClientCert = true }

Managing these entries

consul config write orders-resolver.hcl
consul config read -kind service-resolver -name orders
consul config list -kind service-resolver
consul config delete -kind service-splitter -name orders

Keep every configuration entry in version control and apply it from CI. These files change routing for live traffic, which makes them production code — reviewed, diffed and revertible. A splitter edited by hand in a terminal at 2 a.m. is exactly the change nobody can find the next morning.

Debugging when it does not do what you expect

# what config is actually in effect?
consul config read -kind service-resolver -name orders
consul config read -kind service-splitter -name orders
consul config read -kind service-router   -name orders

# what did Envoy actually receive?
curl localhost:19000/config_dump | jq '.configs[] | select(.["@type"] | contains("Route"))'
curl localhost:19000/clusters
curl localhost:19000/stats | grep -E "upstream_rq_(total|5xx)"

config_dump on the sidecar’s admin port is the authority. Consul’s configuration entry says what you asked for; the Envoy config dump says what is actually routing traffic, and when they disagree the answer is always in the difference.

The three things that are usually wrong:

Protocol is not "http", so the router does nothing.

A subset filter matches nothing — the metadata key is misspelled, or the deploy did not set it. consul catalog services -tags and the health endpoint will show what the instances actually registered.

Split weights do not sum to 100, which is rejected on write, so the old configuration is still live and everything looks unchanged.

What to take from this chapter

• Router → splitter → resolver, in that order. Subsets must exist in a resolver before a splitter can reference them.
• Protocol = "http" is a prerequisite. Without it routers do nothing.
• Subsets are defined by registration metadata, so a deploy places instances in a subset with no Consul change.
• A canary subset should fail over to stable, so a broken canary serves the good version instead of errors.
• Set a request header on the canary path or its errors are 5% of a blended number.
• Retry connection failures freely; retry status codes only where re-execution is safe.
• Rollback is writing the old splitter — seconds, not a deploy.
• Use maglev or ring_hash for session affinity, so scaling does not reshuffle every client.
• config_dump on the sidecar is the authority when configuration and behaviour disagree.

Next: ACLs and security — the token model, the default-deny migration, and the bootstrap you can lock yourself out of.