11. Orchestration and rolling updates

Configuration management makes one machine correct. Orchestration makes a fleet change state in an order that never breaks the service. This is what separates Ansible from the pull-based tools it displaced: an agent waking up every thirty minutes has no way to say “not yet, wait for the other one to finish”.

serial

- hosts: webservers        # 20 hosts
  serial: 2                # two at a time

   serial: 2               2, 2, 2, 2, ... (10 batches)
   serial: '25%'           5, 5, 5, 5
   serial: [1, 5, '50%']   1, then 5, then half the
                           remainder, then the rest
                           ← the canary pattern

The list form is the one to know. serial: [1, 5, '25%'] deploys to one host first, then five, then quarter-batches — so a broken release affects one machine before it affects twenty. Combined with max_fail_percentage, the run stops before the damage spreads.

Two behaviours from chapter 10 to keep in mind: the entire play including handlers completes for one batch before the next starts, and max_fail_percentage is evaluated per batch, not across the play.

The pattern: drain, update, verify, restore

This is the shape of every safe rolling deployment, and it is worth having in full.

- name: Rolling update of the web tier
  hosts: webservers
  become: true
  serial: [1, 3, '25%']
  max_fail_percentage: 0

  pre_tasks:
    - name: Take this host out of the load balancer
      community.general.haproxy:
        state: disabled
        host: "{{ inventory_hostname }}"
        socket: /var/run/haproxy.sock
        backend: web_backend
        wait: true              # wait for connections to drain
      delegate_to: "{{ item }}"
      loop: "{{ groups['loadbalancers'] }}"

    - name: Let in-flight requests finish
      ansible.builtin.pause:
        seconds: 10

  tasks:
    - name: Deploy the new release
      ansible.builtin.unarchive:
        src: "app-{{ app_version }}.tar.gz"
        dest: /opt/app
        remote_src: false

    - name: Restart the application
      ansible.builtin.systemd_service:
        name: app
        state: restarted

  post_tasks:
    - name: Wait for the app to answer locally
      ansible.builtin.uri:
        url: "http://127.0.0.1:8080/healthz"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 30
      delay: 2

    - name: Put this host back in the load balancer
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        socket: /var/run/haproxy.sock
        backend: web_backend
      delegate_to: "{{ item }}"
      loop: "{{ groups['loadbalancers'] }}"

    - name: Confirm the load balancer sees it as healthy
      ansible.builtin.uri:
        url: "http://{{ item }}:8404/stats;csv"
        return_content: true
      register: lb
      delegate_to: localhost
      loop: "{{ groups['loadbalancers'] }}"
      until: "inventory_hostname + ',' in lb.content and ',UP,' in lb.content"
      retries: 10
      delay: 3

Five things that make this correct rather than merely plausible:

wait: true on the drain. Removing a server from a load balancer does not end its existing connections. Without waiting, you restart the application under live traffic — which is the exact failure the drain was supposed to prevent. HAProxy’s chapter on health checks explains what the load balancer is doing on the other side of this.

max_fail_percentage: 0. One host fails, the deployment stops. For a rolling deployment this is almost always what you want — the alternative is continuing to break machines after you have proof the release is bad.

Health check locally before restoring. The application answering on 127.0.0.1 proves it started. Restoring it to the load balancer first and then checking means sending real users to a broken instance.

Confirm from the load balancer’s side too. The last task asks HAProxy whether it considers the backend up. The application being healthy and the load balancer knowing it is healthy are different facts, and the gap between them is however long the health check interval is.

delegate_to with a loop over the load balancers. There is usually more than one, and a host must be re-enabled on all of them.

delegate_to

delegate_to runs a task on a different machine, using the current host’s variables.

- name: Add a DNS record for this host
  community.general.nsupdate:
    server: "{{ dns_server }}"
    zone: example.com
    record: "{{ inventory_hostname_short }}"
    value: "{{ ansible_facts['default_ipv4']['address'] }}"
  delegate_to: localhost

The loop is over web servers, so inventory_hostname and the facts are the web server’s — but the task executes on the control node. That combination is what makes delegation useful: the data comes from one machine, the action happens on another.

delegate_to: localhost           # the control node
delegate_to: "{{ groups['loadbalancers'][0] }}"
delegate_to: db-primary.example.com

# shorthand, older style:
- local_action:
    module: ansible.builtin.uri
    url: https://api.example.com/deploy

delegate_facts: true stores any facts the delegated task gathers against the delegate rather than the current host — which is how you gather facts about a machine that is not in the play:

- ansible.builtin.setup:
  delegate_to: db-primary.example.com
  delegate_facts: true
  run_once: true

run_once

- name: Run the database migration
  ansible.builtin.command: /opt/app/bin/migrate
  run_once: true
  delegate_to: "{{ groups['appservers'][0] }}"

run_once executes the task on one host and applies the result to all of them. The chosen host is the first in the batch — which is why pairing it with delegate_to is better practice: it makes the choice explicit rather than dependent on inventory order.

Two things that catch people:

run_once is per batch, not per play. With serial: 5 across twenty hosts, a run_once task runs four times — once per batch. If it must genuinely happen once, put it in a separate play with hosts: appservers[0], or in a play without serial.

A registered variable from a run_once task is available on every host in that batch, which is the intended and useful behaviour — compute something once, use it everywhere.

throttle and order

- name: Hit a rate-limited API
  ansible.builtin.uri:
    url: "https://api.example.com/register/{{ inventory_hostname }}"
  throttle: 2          # at most 2 hosts on THIS TASK at once

throttle limits one task where serial limits the whole play. It is the right tool when a single step talks to something that cannot take the load — a licence server, an external API — and the rest of the play is fine at full parallelism.

- hosts: webservers
  order: shuffle       # inventory | sorted | reverse_sorted
                       # | reverse_inventory | shuffle

order: shuffle is worth knowing for one reason: with serial, a fixed order means the same host is always the canary, which means it accumulates a different history from the rest. Shuffling spreads that.

Building inventory during the run

Two modules let a play change its own inventory.

- name: Create a VM
  amazon.aws.ec2_instance:
    name: "worker-{{ item }}"
    ...
  register: created
  loop: [1, 2, 3]
  delegate_to: localhost

- name: Add the new machines to this run
  ansible.builtin.add_host:
    name: "{{ item.private_ip_address }}"
    groups: new_workers
    ansible_user: ubuntu
  loop: "{{ created.results | map(attribute='instances') | flatten }}"

- name: Configure them
  hosts: new_workers          # a NEW play — this is required
  roles: [common, worker]

add_host takes effect in the next play, not the current one. A play’s host list is fixed when it starts. This is the standard provision-then-configure shape, and it is the one case where Ansible doing cloud provisioning is genuinely natural — though Terraform’s chapter 15 still makes the case for doing the creation there and the configuration here.

- name: Group hosts by what they are running
  ansible.builtin.group_by:
    key: "os_{{ ansible_facts['distribution'] }}"

- hosts: os_Ubuntu
  tasks: [...]

group_by creates groups from facts, which is a clean alternative to putting when: ansible_facts['distribution'] == 'Ubuntu' on forty tasks.

Waiting for things

- name: Reboot and wait for it to come back
  ansible.builtin.reboot:
    reboot_timeout: 600
    post_reboot_delay: 15
    test_command: systemctl is-system-running --wait

- name: Wait for a port on another machine
  ansible.builtin.wait_for:
    host: "{{ hostvars['db1']['ansible_host'] }}"
    port: 5432
    timeout: 300
  delegate_to: localhost

- name: Wait for the host itself to be reachable by Ansible
  ansible.builtin.wait_for_connection:
    delay: 10
    timeout: 300

wait_for_connection is the one to use after a reboot, because it tests the thing that actually matters — can Ansible connect and run a module — rather than whether a port is accepting TCP. A machine can answer on port 22 well before it can run your playbook.

ansible.builtin.reboot handles the whole dance — issuing the reboot, tolerating the connection dropping, waiting for it to return, and running a test command. Do not write this by hand with shell: reboot and wait_for; the module handles cases you will not think of.

Cross-machine coordination

The pattern for “do something on A, using a value from B”:

- name: Read the primary's replication position
  ansible.builtin.command: >
    psql -tAc "SELECT pg_current_wal_lsn()"
  delegate_to: "{{ groups['db_primary'][0] }}"
  register: lsn
  changed_when: false
  run_once: true

- name: Configure the replicas to start from it
  ansible.builtin.template:
    src: recovery.conf.j2
    dest: /var/lib/postgresql/recovery.conf
  vars:
    start_lsn: "{{ lsn.stdout }}"
  when: inventory_hostname in groups['db_replicas']

This is the kind of thing Ansible is genuinely good at and a pull-based tool cannot express at all. It is also the point where a warning is due: orchestration logic that gets much more complicated than this belongs in a program, not a playbook. If you are writing conditional state machines across machines, chapter 15’s honest assessment applies — Ansible will let you, and you will not enjoy maintaining it.

What to take from this chapter

• serial: [1, 5, '25%'] is the canary pattern: one host, then five, then quarters. The whole play including handlers completes per batch, and max_fail_percentage is evaluated per batch.
• The safe rolling shape is drain → wait → update → health check locally → restore → confirm from the load balancer. Each step exists because skipping it breaks something specific.
• wait: true when draining. Removing a backend does not end its existing connections.
• max_fail_percentage: 0 on deployments. Stop at the first proof the release is bad.
• delegate_to runs a task elsewhere using the current host’s variables — that combination is the whole point. delegate_facts: true gathers facts about a machine not in the play.
• run_once is per batch, not per play. With serial: 5 over twenty hosts it runs four times. Pair it with delegate_to to make the target explicit.
• throttle limits one task; serial limits the play. Use throttle for a rate-limited external service.
• add_host takes effect in the next play, never the current one. group_by turns facts into groups and removes forty when: clauses.
• wait_for_connection after a reboot, not wait_for on port 22 — and use the reboot module rather than writing it yourself.
• Orchestration much more complex than “read a value from A, apply it on B” belongs in a program.

Next: secrets — Ansible Vault, what it does not protect, and how to keep credentials out of the log.