3. The connection and execution model

Chapter 1 said Ansible copies a payload to the target, runs it, and deletes it. This chapter is what that costs, because almost every performance question and a good share of the failure modes come from this one mechanism.

One task, one host

   CONTROL NODE                          TARGET

   read task + args
   render Jinja templates
   build AnsiballZ payload  ─────┐
   (the module, its imports,     │
    and the arguments, in one    │
    self-contained Python file)  │
                                 │
   open SSH connection ──────────┼──────► sshd
                                 │
   mkdir a temp dir ─────────────┼──────► ~/.ansible/tmp/
                                 │        ansible-tmp-1741...
                                 │
   transfer the payload ─────────┴──────► AnsiballZ_copy.py
                                          (sftp, or piped
                                           over the same
                                           connection)

   run: /usr/bin/python3 AnsiballZ_copy.py
                                 ◄─────── {"changed": true,
                                           "dest": "/etc/x",
                                           "mode": "0644"}
   parse the JSON
   delete the temp dir ─────────────────► rm -rf
   report ok / changed / failed

That happens for every task, on every host. Twenty tasks across forty hosts is eight hundred payload transfers and eight hundred Python interpreter starts.

This is the price of agentless. A Puppet agent is already running, already has the catalogue, and needs no SSH round trip per resource. Ansible pays that cost in exchange for having nothing installed anywhere, and the rest of this chapter is about making the price bearable.

AnsiballZ

The payload is worth understanding by name, because it appears in error messages.

AnsiballZ takes the module’s Python source, the source of everything it imports from Ansible’s own library, and the task’s arguments, and packs them into one self-contained Python file with the arguments embedded as a base64 blob. Nothing needs to be installed on the target — no Ansible, no libraries, nothing.

Two consequences:

Modules cannot depend on arbitrary Python packages unless those packages are already on the target. A module that needs boto3 needs boto3 installed on the machine it runs on — which is why cloud modules are almost always run against localhost rather than against the cloud instance.

A traceback mentioning AnsiballZ_something.py is your module failing on the target, not Ansible failing on the control node. That distinction saves a lot of time.

Forks — the default that surprises everyone

# ansible.cfg
[defaults]
forks = 5

Five. Ansible runs each task against five hosts at a time, waits for all five, then takes the next five.

   40 hosts, forks = 5, a task that takes 2 seconds

   ████░ ████░ ████░ ████░ ████░ ████░ ████░ ████░
   └─8 batches × 2s = 16 seconds for ONE task

   forks = 40
   ████████████████████████████████████████
   └─ 2 seconds

This default is from 2012 and it is the first thing to change. Sensible values are 20 to 50 on a laptop, and higher on a dedicated control node. The limits are the control node’s CPU and file descriptors — each fork is a process holding an SSH connection — not the targets, which are doing almost nothing.

ansible-playbook site.yml -f 50

The two settings that matter more than forks

SSH connection reuse

Opening an SSH connection means a TCP handshake, a key exchange, and authentication — typically 100–300 ms. Doing that per task is most of Ansible’s reputation for slowness.

OpenSSH can keep a connection open and multiplex further sessions over it. Ansible turns this on by default, and the default timeout is short:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s

Raise ControlPersist. Sixty seconds is not long enough for a playbook that does slow work between tasks on the same host — the connection closes and the next task pays the full handshake again.

ssh_args = -C -o ControlMaster=auto -o ControlPersist=300s \
           -o PreferredAuthentications=publickey

That second option is worth adding: without it, SSH may try several authentication methods before the one that works, and on a host where GSSAPI or keyboard-interactive is offered first this costs a visible delay on every connection.

Pipelining

By default, Ansible writes the payload to a temp file on the target and then executes it — which means an extra SSH operation per task.

Pipelining sends the payload over the already-open connection to the Python interpreter’s standard input instead. No temp file, no transfer, one fewer round trip.

[ssh_connection]
pipelining = true

This is typically a 30–50% speedup and it is off by default for a historical reason. sudo on old systems was configured with requiretty, which breaks pipelining. On anything modern that setting is gone, and the check is one line:

ansible all -m ansible.builtin.shell -a "grep -r requiretty /etc/sudoers /etc/sudoers.d/ || echo clean"

If that says clean everywhere, turn pipelining on. It is the single best return on effort in this chapter.

Python on the target

Most modules are Python, so the target needs a Python interpreter. Which one is a surprisingly rich source of trouble.

Ansible tries to work it out — it has an ordered list of paths it checks for each distribution — and tells you what it picked:

   [WARNING]: Platform linux on host web1 is using the
   discovered Python interpreter at /usr/bin/python3.9,
   but future installation of another Python interpreter
   could change the meaning of that path.

That warning is telling you something real: discovery can change its answer when the target changes. A machine that gains a newer Python can start running your modules under a different interpreter, with different libraries available.

Pin it in inventory:

group_vars/all.yml

ansible_python_interpreter: /usr/bin/python3

And to silence the warning fleet-wide once you have decided you are happy with discovery:

[defaults]
interpreter_python = auto_silent

Two modules need no Python at all, which is how you bootstrap a machine that does not have it:

- name: Bootstrap Python on a bare host
  ansible.builtin.raw: |
    test -e /usr/bin/python3 || \
      (apt-get update && apt-get install -y python3-minimal)
  changed_when: false

ansible.builtin.raw sends a command straight down the SSH connection with no payload and no JSON. ansible.builtin.script copies a local script and runs it. Both are escape hatches: no idempotence, no check mode, no structured result. Use them to make a host manageable and then never again.

Privilege escalation

Ansible connects as an ordinary user and escalates per task.

- name: Install a package
  ansible.builtin.package:
    name: nginx
    state: present
  become: true
  become_user: root        # the default
  become_method: sudo      # the default; also su, doas,
                           # runas (Windows), pbrun, dzdo

become can be set on a task, a play, a group, or a host. The three things worth knowing:

Escalate per task, not globally, where you can. A play that runs entirely as root will happily do root things you did not intend — and it means every file it creates is owned by root, including ones that should not be.

become plus an unprivileged become_user needs care. Going from deploy to postgres (rather than to root) means the temp directory must be readable by both, and Ansible solves this with ACLs or by making the file world-readable. If allow_world_readable_tmpfiles appears in an error, this is what it is about — the fix is to install acl on the target rather than to enable the setting.

--ask-become-pass (-K) prompts once, and the password is reused for every host. For unattended runs, use passwordless sudo scoped to what the automation actually needs.

Other connections

SSH is the default, not the only option.

   ssh          the default
   local        run on the control node, no SSH at all
   docker       exec into a running container
   podman       the same
   kubectl      exec into a pod
   winrm        Windows, via WinRM
   psrp         Windows, via PowerShell Remoting — newer,
                generally better than winrm
   network_cli  network devices with no shell
   httpapi      devices with a REST API

Two of these change what Ansible is for.

Windows works properly, through winrm or psrp, with a full set of modules in ansible.windows and community.windows. It is not an afterthought, and Ansible is one of very few tools that manages Linux and Windows estates with one language.

Network devices are managed without any agent or Python on the device at all. network_cli logs in and speaks the device’s command line; the modules understand the syntax of Cisco IOS, Juniper Junos, Arista EOS and dozens more. For network teams this is often the entire reason Ansible is in the building, and it is a job containers cannot do at all.

Where configuration comes from

ansible.cfg is found in this order, and the first one found wins completely — there is no merging:

   1. $ANSIBLE_CONFIG            (env var, a file path)
   2. ./ansible.cfg              (current directory)
   3. ~/.ansible.cfg
   4. /etc/ansible/ansible.cfg

A world-writable current directory is skipped, silently, as a security measure — someone could otherwise drop an ansible.cfg in /tmp and change what your next command does. If your settings appear to be ignored and you are working in a shared or oddly-permissioned directory, this is why.

Every setting also has an environment variable, which is what CI usually uses:

ANSIBLE_FORKS=50 ANSIBLE_PIPELINING=true ansible-playbook site.yml

And to find out what is actually in effect:

ansible-config dump --only-changed

That command answers “why is it behaving like this” more often than any other.

What to take from this chapter

• Every task on every host is: build payload → SSH → temp dir → transfer → run Python → parse JSON → delete. That is the price of having no agent.
• AnsiballZ packs the module and its arguments into one self-contained file. A traceback naming AnsiballZ_*.py is the module failing on the target.
• forks defaults to 5. Raise it to 20–50. The limit is your control node, not the targets.
• Raise ControlPersist past its 60-second default, and add PreferredAuthentications=publickey.
• Turn on pipelining. It removes a round trip per task for a 30–50% gain, and the requiretty reason it is off by default is almost certainly gone from your fleet — one grep confirms it.
• Pin ansible_python_interpreter. Discovery can silently change its answer when a target gains a new Python.
• raw and script need no Python and are how you bootstrap a bare host. They have no idempotence and no check mode — use them once.
• Escalate per task rather than per play, and if allow_world_readable_tmpfiles appears in an error, install acl on the target instead of enabling it.
• Windows via psrp and network devices via network_cli are first-class, and are jobs containers cannot do.
• The first ansible.cfg found wins entirely, and a world-writable directory is skipped silently. ansible-config dump --only-changed tells you what is really in effect.

Next: playbooks — the structure of a play, the four kinds of task section, and handlers, which do not run when you think they do.