Skip to content
Toolforge

12. Connections, variables and secrets

Airflow needs credentials to almost everything it touches. Where those credentials live is one of the more consequential decisions in an installation, and the default is weaker than most people assume.

Connections

Section titled “Connections”

A Connection is a named set of credentials for an external system.

Terminal window
airflow connections add warehouse \
  --conn-type postgres \
  --conn-host db.internal --conn-port 5432 \
  --conn-login airflow --conn-password 's3cr3t' \
  --conn-schema analytics \
  --conn-extra '{"sslmode": "require"}'
@task
def load():
    hook = PostgresHook(postgres_conn_id="warehouse")     # ← by name
    hook.run("...")

The point is indirection. The DAG names a connection; it does not contain credentials. Rotate the password and no code changes.

Connections can also come from the environment, which is how they are usually supplied in containers:

Terminal window
AIRFLOW_CONN_WAREHOUSE='postgresql://airflow:s3cr3t@db.internal:5432/analytics?sslmode=require'

Environment connections take precedence over database ones and are invisible in the UI — which is either good hygiene or a confusing surprise, depending on whether you knew.

Variables

Section titled “Variables”

Key-value configuration:

Terminal window
airflow variables set batch_size 5000
airflow variables set warehouse_config '{"schema": "prod", "retries": 3}'
# ✗ at module level — a DATABASE QUERY EVERY 30 SECONDS (chapter 3)
BATCH = Variable.get("batch_size")


# ✓ in a template — fetched at run time
BashOperator(bash_command="./run.sh --batch {{ var.value.batch_size }}")


# ✓ inside a task
@task
def run():
    batch = Variable.get("batch_size")

The chapter-3 rule in its most common violation. Every Variable.get() at module level is a query on every parse, on the processor and on every worker.

If you need many values, fetch one JSON variable rather than ten:

cfg = Variable.get("warehouse_config", deserialize_json=True)   # one query

Variables are not secrets. They are visible in the UI to anyone who can log in. Airflow masks values whose key contains password, secret, token, key, api and similar — which is a UI nicety, not a security boundary, and easily defeated by naming something db_creds.

The Fernet key, and what it does not do

Section titled “The Fernet key, and what it does not do”

Connection passwords and variable values are encrypted at rest in the metadata database with a symmetric Fernet key:

[core]
fernet_key = ZmDf...=

What this protects: someone who gets a database dump and not the key.

What it does not protect against — and this is the part that matters:

Anyone who can run a DAG. Task code can call Connection.get() or Variable.get() and print the result. The Fernet key is available to the process that needs to decrypt, which is the process running your code.

Anyone with UI admin rights. They can view and edit connections.

Anyone who has both the database and the key. Which, in most deployments, is the same Kubernetes namespace.

The Fernet key protects a stolen backup. It does not create a boundary between DAGs.

And in Airflow 2 there was no such boundary at all: every worker held metadata database credentials, so any task in any DAG could read every connection in the installation. A DAG belonging to the marketing team could read the production database password.

Airflow 3’s Task Execution API (chapter 2) is what fixes this. Tasks request what they need through the API server with a scoped token, and never touch the database. If you run a multi-team Airflow on version 2, this is your most important upgrade reason.

Rotating the Fernet key is possible and awkward:

fernet_key = new_key,old_key      # new first, old still able to decrypt
Terminal window
airflow rotate-fernet-key         # re-encrypt everything with the new key

Then remove the old one. Losing the Fernet key means every stored credential is unrecoverable and must be re-entered by hand.

Secrets backends

Section titled “Secrets backends”

The better answer: do not store secrets in Airflow at all.

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {
  "connections_path": "airflow/connections",
  "variables_path": "airflow/variables",
  "mount_point": "secret",
  "url": "https://vault.internal:8200",
  "auth_type": "kubernetes",
  "kubernetes_role": "airflow"
}

Now PostgresHook(postgres_conn_id="warehouse") looks up secret/airflow/connections/warehouse in Vault. Nothing is in Airflow’s database.

The lookup order:

   1. the configured secrets backend
   2. environment variables  (AIRFLOW_CONN_*, AIRFLOW_VAR_*)
   3. the metadata database

Available backends include Vault, AWS Secrets Manager, AWS SSM Parameter Store, GCP Secret Manager, Azure Key Vault, and a local file backend for development.

What you gain:

Rotation without touching Airflow. Change it in the store; the next task picks it up.

Central audit. Vault records who read which secret and when — including Airflow.

One store for the whole organisation, not one per tool.

Dynamic credentials, if the store supports them. Vault’s database engine can issue a credential that did not exist before the task asked and is destroyed an hour later. An Airflow task that reads a database with a credential nobody has ever seen is a materially better position than a shared password encrypted at rest.

The one cost to plan for: every connection lookup is a network call. With hundreds of tasks that is real traffic, so enable caching:

[secrets]
use_cache = True
cache_ttl_seconds = 900

And know the failure mode: if the secrets backend is down, tasks cannot get credentials and fail. That is a new dependency on your critical path. It is the right trade, and it should be a deliberate one.

Params

Section titled “Params”

Per-run inputs, distinct from Variables:

from airflow.sdk import Param


@dag(
    schedule=None,
    params={
        "region": Param("eu-west-1", type="string",
                        enum=["eu-west-1", "us-east-1"]),
        "dry_run": Param(True, type="boolean"),
        "limit": Param(1000, type="integer", minimum=1, maximum=100000),
    },
)
def manual_export():
    @task
    def run(**context):
        region = context["params"]["region"]

Params are validated against a JSON schema and rendered as a form when someone triggers the DAG manually. This is the right way to build an operator-triggered job — a typed, validated form instead of a wiki page explaining which Variable to edit first.

What goes where

Section titled “What goes where”
   CONNECTION   credentials for an external system
                → and put them in a SECRETS BACKEND


   VARIABLE     installation-wide configuration that
                changes without a deploy
                → not secrets. Fetch at run time, not
                  at parse time.


   PARAM        per-run input, validated, form-rendered
                → for manually triggered DAGs


   ENV VAR      static configuration baked into the
                deployment
                → the AIRFLOW__ prefix; simplest thing
                  that works


   CODE         anything that should be reviewed
                → most "configuration" belongs here

That last line is worth stating as a position. A great deal of what ends up in Airflow Variables should be in the repository. A batch size, a list of tables, a threshold — changing those in a UI is an unreviewed production change with no history and no rollback. It is exactly the argument the Argo CD guide makes about clicking changes into a cluster, and it applies here for the same reasons.

Keep Variables for things that must genuinely change without a deployment: a feature flag, a manual kill switch, a rotating endpoint.

What to take from this chapter

Section titled “What to take from this chapter”
  • Connections provide indirection: DAGs name credentials rather than containing them. AIRFLOW_CONN_* environment variables take precedence and are invisible in the UI.
  • Never call Variable.get() at module level — it is a database query every 30 seconds on every parser and worker. Use {{ var.value.x }} or fetch inside the task; one JSON variable beats ten.
  • Variables are not secrets. UI masking is a nicety, not a boundary.
  • The Fernet key protects a stolen backup only. It does not create a boundary between DAGs — task code can decrypt anything it can read.
  • In Airflow 2, any task could read every connection in the installation. Airflow 3’s Task Execution API fixes this and is the main upgrade argument for multi-team installations.
  • Losing the Fernet key means re-entering every credential. Rotation is new,old then airflow rotate-fernet-key.
  • Use a secrets backend — Vault, AWS/GCP/Azure. Lookup order is backend → environment → database. You gain rotation, audit, one store, and the option of dynamic credentials.
  • Enable secrets caching, and accept that the backend is now on your critical path.
  • Params are typed, validated, form-rendered per-run inputs — the right way to build a manually triggered job.
  • Most “configuration” belongs in the repository, not in Variables. A UI edit is an unreviewed production change with no history.

Next: tuning and operations — the concurrency knobs, pools, deployment patterns, monitoring and upgrades.

Written by Sepuri Sai Krishna

© 2026CC BY 4.0Source