12. Secrets and Ansible Vault

Ansible connects as root to every machine you own. It needs database passwords, API keys, TLS private keys and certificates. All of it lives in a git repository.

Ansible Vault encrypts files and values in place, so the repository holds ciphertext and the decryption key lives somewhere else.

The commands

ansible-vault create   group_vars/production/vault.yml
ansible-vault edit     group_vars/production/vault.yml
ansible-vault view     group_vars/production/vault.yml
ansible-vault encrypt  existing-file.yml
ansible-vault decrypt  file.yml            # back to plaintext
ansible-vault rekey    file.yml            # change the password

An encrypted file is AES-256 and looks like this:

$ANSIBLE_VAULT;1.1;AES256
36396266383564363733353066363134323065616434653237303566623336
39373466613463623739656334323938343738623333366634636234626434
...

Git sees a text file that changes completely on every edit. That is worth knowing: encrypted files do not diff, do not merge, and produce a conflict whenever two people edit them. Keep them small and split by concern.

Running a playbook that needs one:

ansible-playbook site.yml --ask-vault-pass
ansible-playbook site.yml --vault-password-file ~/.vault_pass

# ansible.cfg
[defaults]
vault_password_file = ~/.vault_pass

A password file must not be in the repository. Put it in your home directory, chmod 600, and add the path to .gitignore as a second line of defence.

It can also be a script that prints the password, which is how CI does it without a file on disk:

#!/bin/sh
# ~/.vault_pass_from_env.sh  — chmod +x
printf '%s' "$ANSIBLE_VAULT_PASSWORD"

Ansible detects the executable bit and runs it instead of reading it.

The structure that works

Do not encrypt whole variable files. Split plain from secret:

   group_vars/
     production/
       main.yml       ← plain, readable, diffable
       vault.yml      ← encrypted

# main.yml — visible in every diff and code review
db_host: db.example.com
db_port: 5432
db_user: app
db_password: "{{ vault_db_password }}"      # ← indirection

# vault.yml — encrypted
vault_db_password: "hunter2-but-better"
vault_api_token: "sk_live_..."

Prefix every variable inside a vault file with vault_, and reference it indirectly from a plain file. This buys two real things:

You can see what secrets exist without decrypting anything. grep vault_ group_vars/*/main.yml lists every secret the estate uses, and a code review shows that db_password is set from a vault variable even though the value is invisible.

A missing secret fails clearly. Without indirection, a role referring to db_password on a host whose vault file was never created gets an undefined-variable error naming a variable that appears nowhere in the repository, which is a confusing thing to debug.

Encrypting a single value

You do not have to encrypt a whole file:

ansible-vault encrypt_string 'sk_live_abc123' --name 'vault_api_token'

vault_api_token: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  62313365396662343061393464336163383764373764613633653634306232
  ...

That can sit inside an otherwise plaintext file. Use it sparingly. A file with three inline vault strings among forty plain values is hard to scan and easy to add a plaintext secret to by accident. A separate vault.yml makes the boundary obvious.

Vault IDs

One password for everything means everyone who can deploy to staging can decrypt production. Vault IDs fix that:

ansible-vault encrypt --vault-id prod@prompt   group_vars/production/vault.yml
ansible-vault encrypt --vault-id stage@prompt  group_vars/staging/vault.yml

ansible-playbook site.yml \
  --vault-id stage@~/.vault_stage \
  --vault-id prod@~/.vault_prod

[defaults]
vault_identity_list = stage@~/.vault_stage, prod@~/.vault_prod

The ID is stored in the file header, so Ansible knows which password to try. Supply only the staging ID and production files simply cannot be read.

This is the minimum for any team with more than one environment, and it is one command more work than not doing it.

The four ways a secret leaks anyway

Vault encrypts a file. It does nothing about what happens to the value once it is decrypted, and this is where secrets actually escape.

1. Task output

- ansible.builtin.uri:
    url: https://api.example.com/deploy
    headers:
      Authorization: "Bearer {{ vault_api_token }}"
  no_log: true            # ← without this, -v prints the token

no_log: true on every task that touches a secret. Ansible redacts obvious password arguments in some modules, and it does not catch headers, command lines, template contents or registered results. Verbose output, a failure message, or a CI log then contains the secret permanently.

The cost is that debugging becomes harder — the task prints censored. Toggle it temporarily rather than leaving it off.

2. Loops

- ansible.builtin.user:
    name: "{{ item.name }}"
    password: "{{ item.hash }}"
  loop: "{{ users }}"
  no_log: true
  loop_control:
    label: "{{ item.name }}"     # ← and this

A loop prints its entire item by default, so a list of dictionaries containing passwords prints all of them, on every iteration, even at default verbosity. Chapter 8 made this point about readability; it is also a security control.

3. The command line

- ansible.builtin.command: "mysql -u root -p{{ vault_mysql_root }}"

The password is now in the target’s process list, visible to every user on that machine for as long as the command runs, and in the shell history if anyone reproduces it. Use a module that takes the credential as an argument, or write a temporary credentials file with mode: '0600' and remove it afterwards.

4. Rendered files

A template that writes a secret into /etc/app/config.yml leaves it there in plaintext. That is often unavoidable — the application has to read it — but:

- ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    owner: app
    group: app
    mode: '0600'          # not 0644. Not the default.
  no_log: true

Set mode explicitly on anything containing a secret. The default is influenced by the remote user’s umask, and 0644 on a file with a database password means every user on that machine can read it.

Fetching from a real secret store

Vault-the-Ansible-feature is file encryption. Vault-the-HashiCorp-product is a secret management system, and they solve different problems. Beyond a certain size you want the second one.

- name: Get the database password at run time
  ansible.builtin.set_fact:
    db_password: "{{ lookup('community.hashi_vault.hashi_vault',
                     'secret=secret/data/prod/db:password') }}"
  no_log: true

   ANSIBLE VAULT                 A SECRET STORE
                                 (HashiCorp Vault, AWS
                                  Secrets Manager, etc)

   secret lives in git           secret lives in the store
   rotation = edit, commit,      rotation = change it once,
     deploy everywhere             no repository change
   no audit trail                every read is audited
   no expiry                     leases and TTLs
   revocation = rotate and       revocation = immediate
     redeploy
   one password unlocks a        per-identity access
     whole environment

The decision point is usually rotation. If a leaked credential means editing files, committing, reviewing and redeploying, you will not rotate often — and the Vault guide’s first chapter makes exactly this argument. Its dynamic secrets go further: a credential created for this playbook run, valid for an hour, that nobody needs to rotate because it expires on its own.

Ansible Vault remains right for a lot of cases — a small team, a handful of secrets, no compliance requirement for audit trails. It is enormously better than plaintext in git, which is the alternative it is actually competing with. But it does not scale, and knowing what it does not do is the point of this section.

The middle ground worth knowing:

   Ansible Vault           small teams, few secrets, no audit
                           requirement

   SOPS + KMS/age          encrypted files, but decryption
                           is granted by cloud IAM rather
                           than a shared password. Audited
                           at the KMS.

   HashiCorp Vault / AWS   nothing in the repository,
     Secrets Manager       audited, rotatable, leased

SOPS is the most under-considered of these. It keeps the “encrypted file in git” model that makes Ansible Vault convenient, and replaces the shared password with cloud IAM — so access is per-person, revocable, and logged, and there is no password to leak.

Protecting the control node

The last point, and the one most often missed.

The control node holds credentials that can reach everything. SSH keys, the vault password, cloud credentials, sudo rights on the entire estate. It is the single most valuable machine you own, and it is frequently somebody’s laptop.

   ✓ SSH keys with passphrases, in an agent
   ✓ the vault password file at 0600, never in the repo
   ✓ run production playbooks from CI or a bastion,
     not from laptops
   ✓ short-lived SSH certificates rather than
     permanent authorized_keys, if you can
   ✓ MFA on whatever holds the vault password
   ✗ do not commit .vault_pass. Add it to .gitignore
     in the same commit that creates the repository.

And a .gitignore entry that has saved people:

   *.pem
   *_rsa
   .vault_pass*
   *.decrypted.yml

That last one matters because ansible-vault decrypt writes plaintext in place — the encrypted file becomes a plaintext file with the same name. Decrypt, forget to re-encrypt, commit, and the secret is in the history forever. Use ansible-vault edit, which decrypts to a temporary file and re-encrypts on save, and never decrypt.

What to take from this chapter

Ansible Vault encrypts files and single values with AES-256, so a repository holds ciphertext. Encrypted files do not diff or merge — keep them small.
Split plain from secret: main.yml referencing {{ vault_db_password }}, vault.yml holding it. Prefix vault variables with vault_ so you can see what exists without decrypting.
Use vault IDs so staging access does not imply production access. One extra flag.
Four leaks: task output (no_log: true), loops (loop_control: label), the command line (process list), and rendered files (set mode: '0600').
no_log is required for headers, command lines and registered results — Ansible’s automatic redaction does not cover them.
Ansible Vault versus a secret store is a question about rotation. If rotating means edit-commit-review-deploy, you will not do it. SOPS with cloud KMS is the under-considered middle ground — files in git, but access by IAM rather than a shared password.
The control node can reach everything. Run production from CI or a bastion, not from a laptop.
Never ansible-vault decrypt — it writes plaintext in place under the same name. Use edit.

Next: testing, tuning and operations — check mode in CI, Molecule, the idempotence test, and the settings that make a slow playbook fast.

Written by Sepuri Sai Krishna