10. A play traced end to end

Here is a small playbook run against three web servers, followed step by step. The point is not the playbook; it is knowing exactly what happens between pressing enter and seeing the recap.

---
- name: Configure web tier
  hosts: webservers
  become: true
  serial: 2

  tasks:
    - name: Ensure nginx is installed
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Deploy the site configuration
      ansible.builtin.template:
        src: site.conf.j2
        dest: /etc/nginx/conf.d/site.conf
        validate: 'nginx -t -c %s'
      notify: Reload nginx

    - name: Ensure nginx is running
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

  handlers:
    - name: Reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

ansible-playbook -i inventory/production playbooks/web.yml

Phase 1 — before anything touches a machine

1. Find the configuration

Ansible resolves ansible.cfg in the order from chapter 3 — $ANSIBLE_CONFIG, then ./ansible.cfg, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg — and the first one found wins entirely. Forks, pipelining, roles_path and everything else are now fixed for this run.

2. Parse the playbook

The YAML is read and validated. Every import_tasks, import_role and import_playbook is expanded now, recursively, before a single host is contacted — this is what “static” means in chapter 8. include_* statements are left as tasks to be dealt with later.

A YAML syntax error stops here, with nothing having run. So does a missing imported file.

3. Build the inventory

The inventory source is parsed — a file, a directory, or a dynamic plugin making API calls. Groups are resolved, children are flattened, and the full host list is built.

Then webservers is matched against it, and --limit narrows the result further. If nothing matches:

   [WARNING]: Could not match supplied host pattern, ignoring: webservers
   PLAY [Configure web tier] ****
   skipping: no hosts matched

That is a warning, not an error, and the exit code is 0. A typo in a group name produces a run that succeeds and does nothing — which is chapter 14’s material and a good reason for CI to assert that hosts were actually touched.

4. Merge variables

For each host, all twenty-two precedence levels from chapter 6 are merged into one dictionary: role defaults, group_vars/all, each group in turn, host_vars, play vars, extra vars. Nothing is evaluated yet — {{ }} expressions are stored as strings and rendered lazily, at the moment each one is used.

That laziness matters. A variable referring to a fact that does not exist yet is fine as long as nothing reads it before facts are gathered.

Phase 2 — the play begins

5. Batching

serial: 2 splits the three hosts into batches:

   batch 1:  web1, web2
   batch 2:  web3

The entire play runs for batch 1 — all tasks and all handlers — before batch 2 starts. This is chapter 11’s subject and the mechanism behind every rolling update.

6. Fact gathering

   TASK [Gathering Facts] ***************************
   ok: [web1.example.com]
   ok: [web2.example.com]

This is the setup module, and it is a real task — a full connection, payload transfer and Python execution on every host, collecting several hundred facts. On a slow or distant machine this is often the single most expensive step in a short playbook.

With gathering = smart and a fact cache, hosts whose facts are already cached are skipped entirely.

Phase 3 — the per-task loop

For each task, for every host in the batch, up to forks at a time:

7. Evaluate when

Conditions are rendered against that host’s merged variables. False means skipped, and the host does nothing for this task. serial batching and when are independent — a skip does not remove the host from the play.

8. Render the arguments

Every argument passes through Jinja (chapter 7) using that host’s variables. dest: /etc/nginx/conf.d/site.conf is trivial; src: site.conf.j2 for the template module means the file is read on the control node and rendered there, with the result carried to the target.

9. Build the payload

AnsiballZ (chapter 3) packs the module source, its Ansible library imports, and the rendered arguments into one Python file.

10. Connect

   first task on this host:
     TCP handshake + key exchange + auth   ~150 ms
     → a ControlPersist socket is left open

   every later task on this host:
     reuse the socket                       ~2 ms

This is why ControlPersist matters so much. The first task pays; the rest do not, as long as the socket survives.

11. Escalate and execute

   with pipelining OFF (the default):
     mkdir  ~/.ansible/tmp/ansible-tmp-1741.../
     sftp   AnsiballZ_package.py  →  that directory
     exec   sudo -H -S -n /usr/bin/python3 <that file>
     rm -rf the directory

   with pipelining ON:
     exec   sudo -H -S -n /usr/bin/python3
     ...and write the payload to its stdin
     (no temp file, no transfer, one fewer round trip)

12. The module runs on the target

For package:

   detect the package manager (apt here)
   ask dpkg-query whether nginx is installed
     → not installed
   run apt-get install -y nginx
   print: {"changed": true, "stdout": "...", ...}
   exit 0

For template, with validate:

   receive the rendered content
   write it to a TEMPORARY path
   run: nginx -t -c <that temp path>
     → exits 0
   compare against the existing file
     → different
   move the temp file into place atomically
   print: {"changed": true, "diff": {...}}

Note the order: validate first, then move. A broken template fails the task with the old file still in place (chapter 4).

13. Read the result

The control node parses the single JSON line, decides ok / changed / failed, prints it, and — if changed and the task has notify — records the handler notification for this host. The handler does not run.

   TASK [Ensure nginx is installed] *****************
   changed: [web1.example.com]
   changed: [web2.example.com]

   TASK [Deploy the site configuration] *************
   changed: [web1.example.com]
   changed: [web2.example.com]
        → "Reload nginx" notified on both

   TASK [Ensure nginx is running] *******************
   changed: [web1.example.com]
   changed: [web2.example.com]

14. Failure handling

If a module returns failed, that host is removed from the rest of the play. Other hosts continue. Unless:

   any_errors_fatal: true       → the whole play stops now
   max_fail_percentage exceeded → the play aborts after
                                  this task
   the task is in a block with
     a rescue                   → the rescue runs instead

Phase 4 — handlers

Once the last task in the batch completes:

   RUNNING HANDLER [Reload nginx] *******************
   changed: [web1.example.com]
   changed: [web2.example.com]

Handlers run in the order they are defined, once each, only on hosts that notified them, and only if the host has not failed — unless force_handlers is set (chapter 4).

Then batch 1 is finished, and the whole play repeats from step 6 for batch 2: fact gathering, every task, handlers.

Phase 5 — the recap

PLAY RECAP ***************************************
web1.example.com : ok=5  changed=4  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web2.example.com : ok=5  changed=4  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web3.example.com : ok=5  changed=4  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

The exit code:

   0   everything succeeded
   1   an error occurred
   2   one or more hosts failed
   4   one or more hosts were unreachable
   99  the run was interrupted
   250 an unexpected internal error

2 and 4 are different and CI should treat them differently. A failed host means your automation is wrong; an unreachable host often means a machine is down, which is a different alert entirely.

Where the time actually goes

For a three-task play across fifty hosts:

   inventory + parsing            0.5 s      once
   fact gathering                 2-10 s     per host, parallel
   connection setup               0.15 s     per host, first task only
   per task, per host:
     render + build payload       ~5 ms      control node CPU
     transfer                     ~20 ms     removed by pipelining
     python startup on target     50-200 ms  ← usually the largest
     the actual work              varies
     teardown                     ~10 ms

   → the floor for ANY task is roughly
     100 ms × ceil(hosts / forks)

Python interpreter startup on the target is usually the biggest per-task cost, and nothing in your control removes it — which is why the levers that matter are the ones that reduce the number of round trips and increase parallelism: more forks, pipelining on, facts cached, and fewer tasks doing more each (chapter 8’s “pass the list to package” is exactly this).

Measure it rather than guessing:

[defaults]
callbacks_enabled = ansible.posix.profile_tasks

   Deploy the site configuration -------------- 12.44s
   Gathering Facts ---------------------------- 8.01s
   Ensure nginx is installed ------------------ 3.12s

Almost every slow playbook has one or two tasks that dominate, and this is how you find them in thirty seconds.

What to take from this chapter

• Before any machine is touched: config resolved (first file wins), playbook parsed, all import_* expanded, inventory built, variables merged. Templates render lazily, at use.
• A host pattern that matches nothing is a warning with exit code 0. A typo runs successfully and does nothing.
• serial runs the entire play — tasks and handlers — for one batch before the next begins.
• Fact gathering is a real task with a full round trip per host, and often the largest cost in a short play.
• Per task per host: evaluate when → render arguments → build AnsiballZ → connect (reused after the first) → escalate → run → parse JSON → record notifications.
• template renders on the control node; validate runs on the target before the file is moved into place.
• A failed host is removed from the rest of the play while others continue — unless any_errors_fatal, max_fail_percentage, or a rescue.
• Handlers run at the end of each batch, in definition order, only on hosts that notified and did not fail.
• Exit codes distinguish 2 (host failed) from 4 (host unreachable) — CI should too.
• Python startup on the target dominates per-task cost. The levers are fewer round trips and more parallelism. profile_tasks finds the slow ones in thirty seconds.

Next: orchestration — rolling updates, draining a load balancer, delegation, and running something exactly once across a fleet.