13. Testing, tuning and operations

Testing, in four layers

Each layer catches something the one before it cannot, and each costs more to run.

   1. ansible-playbook --syntax-check      milliseconds
   2. ansible-lint                          seconds
   3. --check --diff against real hosts     a minute
   4. Molecule: converge in a container,    minutes
      then run again and assert nothing
      changed

The idempotence test

The most valuable test in Ansible is also the simplest:

ansible-playbook site.yml            # first run
ansible-playbook site.yml            # second run — expect changed=0

A second run that reports changes has found a bug. Either a task is not idempotent (chapter 5’s command problem), or it is fighting something else on the machine, or a lineinfile regex matches its own output.

Automate the assertion:

ansible-playbook site.yml | tee /tmp/second.log
grep -E 'changed=[1-9]' /tmp/second.log && {
  echo "NOT IDEMPOTENT"; exit 1; }

Molecule does this for you, and it is worth doing by hand in CI even without Molecule.

ansible-lint

pip install ansible-lint
ansible-lint
ansible-lint --write            # auto-fix what it can

It catches the things this guide has warned about, mechanically:

   command-instead-of-module    use `package`, not `apt-get`
   no-changed-when              a command with no changed_when
   risky-file-permissions       a file created with no mode
   fqcn                         short module names
   no-free-form                 `command: foo bar` instead of
                                the structured form
   name[missing]                a task with no name
   risky-shell-pipe             a shell pipe without pipefail
   ignore-errors                flagged for review
   partial-become               become_user without become
   var-naming                   invalid or ansible_-prefixed names

.ansible-lint

profile: production      # min | basic | moderate | safety
                         # | shared | production
exclude_paths:
  - galaxy_roles/
  - collections/
skip_list:
  - yaml[line-length]

profile: production is the strict setting and is the right thing to aim at. Start at basic, fix everything, and move up one profile at a time — turning on the strictest profile against an existing codebase produces a thousand findings and gets switched off again.

Molecule

Molecule runs a role against a real container, then verifies it.

pip install molecule molecule-plugins[docker]
molecule init scenario -d docker
molecule test

molecule/default/molecule.yml

driver:
  name: docker
platforms:
  - name: ubuntu2204
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
    command: /lib/systemd/systemd
    privileged: true
    cgroupns_mode: host
    volumes: ['/sys/fs/cgroup:/sys/fs/cgroup:rw']
  - name: rocky9
    image: geerlingguy/docker-rockylinux9-ansible:latest
    pre_build_image: true
verifier:
  name: ansible

molecule test runs the full sequence: lint, create, converge, idempotence, verify, destroy.

molecule/default/verify.yml

- name: Verify
  hosts: all
  tasks:
    - ansible.builtin.service_facts:
    - ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'

    - ansible.builtin.uri:
        url: http://localhost/
        status_code: 200

The systemd-in-Docker configuration above is the fiddly part, and it is why the geerlingguy pre-built images are worth using — they are set up for exactly this. Without a working init system, any role that manages services cannot be tested in a container at all.

Molecule is worth it for roles you publish or share. For a playbook that configures your own three servers, --check --diff plus an idempotence run in CI gets you most of the value for a fraction of the setup.

Tags

- ansible.builtin.package: {name: nginx, state: present}
  tags: [nginx, packages]

- ansible.builtin.assert:
    that: [app_version is defined]
  tags: [always]              # runs even when --tags is used

ansible-playbook site.yml --tags nginx
ansible-playbook site.yml --tags "nginx,ssl"
ansible-playbook site.yml --skip-tags packages
ansible-playbook site.yml --list-tags
ansible-playbook site.yml --list-tasks --tags nginx

   always      runs unless explicitly skipped
   never       runs only if requested by name
   tagged      any task with any tag
   untagged    any task with none
   all         everything (the default)

never is the useful obscure one:

- name: Wipe and rebuild the database
  ansible.builtin.command: /opt/app/bin/reset-db
  tags: [never, destructive]

It cannot run by accident. Only --tags destructive reaches it.

Two rules that keep tags usable:

Tag at the role or block level, not per task. Fifty individually tagged tasks is unmaintainable; a block with one tag is clear.

Remember chapter 8’s trap: tags do not propagate into include_* without apply:, but they do into import_*.

Making it fast

The settings, in order of return.

# ansible.cfg
[defaults]
forks = 50                     # from 5. Biggest single win.
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_fact_cache
fact_caching_timeout = 7200
callbacks_enabled = ansible.posix.profile_tasks

[ssh_connection]
pipelining = true              # 30-50%, one setting
ssh_args = -C -o ControlMaster=auto -o ControlPersist=600s \
           -o PreferredAuthentications=publickey
control_path = /tmp/.ansible-%%h-%%p-%%r

Beyond configuration, five things in the playbooks themselves:

Pass lists to modules instead of looping (chapter 8). One package call with fifty names, not fifty calls.

Turn off fact gathering where you do not need facts. gather_facts: false on a play that only restarts a service saves a full round trip per host.

Gather a subset when you need only some:

gather_subset: ['!all', '!min', 'network']

Do not put command in a loop over anything large. Each iteration is a full payload, transfer and Python start. Build the work into one script, or use a module that takes a list.

Use async for genuinely long tasks:

- name: Run the long import
  ansible.builtin.command: /opt/app/bin/import-everything
  async: 3600          # allow up to an hour
  poll: 0              # fire and forget; do not wait
  register: import_job

- name: ...other work happens here...

- name: Wait for the import
  ansible.builtin.async_status:
    jid: "{{ import_job.ansible_job_id }}"
  register: result
  until: result.finished
  retries: 120
  delay: 30

poll: 0 also survives a dropped SSH connection, which is the other reason to use it — a task that takes forty minutes over a flaky link will otherwise fail partway through with the work still running on the target.

Finding the slow part

callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer

   Gathering Facts ------------------------- 41.03s
   Install base packages ------------------- 22.14s
   Deploy configuration --------------------- 3.02s
   ...
   Playbook run took 0 days, 0 hours, 2 minutes, 11 seconds

Almost every slow playbook is one or two tasks. Measure before tuning; the answer is usually fact gathering (cache it) or a loop that should have been a list.

Output and logging

[defaults]
stdout_callback = ansible.builtin.yaml     # readable multi-line
                                           # output instead of
                                           # one long JSON line
log_path = /var/log/ansible.log
display_skipped_hosts = false
display_ok_hosts = false                   # show only changes

stdout_callback = yaml is a quality-of-life change worth making immediately. The default prints a failed task’s result as one enormous JSON line; the YAML callback prints it as readable, indented output with real newlines in stdout.

Verbosity:

   -v      task results
   -vv     task paths, and which file each came from
   -vvv    connection details, the commands being run
   -vvvv   connection plugin debugging, SSH itself

-vv is the useful everyday level — it tells you which file a task came from, which answers most “why is this running” questions on its own.

Running it from CI

# .gitlab-ci.yml, or the equivalent
lint:
  script:
    - ansible-lint
    - ansible-playbook --syntax-check playbooks/site.yml

check:
  script:
    - ansible-playbook -i inventory/production playbooks/site.yml
        --check --diff
  # informational: check mode has false positives (chapter 5)

deploy:
  when: manual
  script:
    - ansible-playbook -i inventory/production playbooks/site.yml
  environment: production

Four things that make CI runs behave:

export ANSIBLE_HOST_KEY_CHECKING=False   # ephemeral runners
export ANSIBLE_FORCE_COLOR=1             # readable logs
export ANSIBLE_STDOUT_CALLBACK=yaml
export ANSIBLE_PIPELINING=true

ANSIBLE_HOST_KEY_CHECKING=False is a real trade. It removes host key verification, which means a machine-in-the-middle can impersonate your servers. On a fresh runner with no known_hosts there is no good alternative short of distributing a known_hosts file — and doing that is the better answer if the environment justifies it.

Check the exit code properly. Chapter 10’s codes: 2 is a failed host, 4 is unreachable. Treat them differently — unreachable usually means a machine is down, not that your automation is wrong.

Ad-hoc commands worth keeping

# is everything alive?
ansible all -m ansible.builtin.ping

# what version of a package is out there?
ansible all -m ansible.builtin.package_facts \
  -a "manager=auto" | grep -A2 openssl

# a fact across the fleet
ansible all -m ansible.builtin.setup \
  -a "filter=ansible_distribution_version"

# disk space
ansible all -m ansible.builtin.shell -a "df -h /" -o

# who is logged in
ansible all -m ansible.builtin.command -a "who" -o

# emergency: restart a service everywhere
ansible webservers -m ansible.builtin.systemd_service \
  -a "name=nginx state=restarted" --become --limit 'webservers[0:2]'

-o gives one line per host, which makes a fleet-wide command readable. And note --limit on the last one: use it on ad-hoc commands too, because there is no --check safety net for a systemd_service restart.

What to take from this chapter

• Four layers: --syntax-check, ansible-lint, --check --diff, Molecule. Each catches what the previous cannot.
• The idempotence test — run twice, expect changed=0 — is the most valuable test there is. Assert it in CI.
• ansible-lint with profile: production mechanically catches most of this guide’s warnings. Start at basic and move up a profile at a time.
• Molecule is worth it for shared roles; for your own three servers, check mode plus an idempotence run gets most of the value.
• Tag at the role or block level. tags: [never] protects destructive tasks from ever running by accident; tags: [always] keeps assertions running under --tags.
• Speed, in order: forks = 50, pipelining = true, fact caching, ControlPersist=600s, then pass lists instead of looping.
• async with poll: 0 for long tasks — it also survives a dropped connection.
• profile_tasks finds the one or two tasks that are the whole runtime.
• stdout_callback = yaml immediately; -vv is the useful everyday verbosity.
• In CI: know that ANSIBLE_HOST_KEY_CHECKING=False really does remove host verification, and treat exit code 2 differently from 4.
• Use --limit on ad-hoc commands. There is no check mode to save you.

Next: failure modes — the twelve things that actually go wrong, and what each one looks like.