5. The reconciliation loop traced

This is the chapter to come back to when someone says “I pushed twenty minutes ago and nothing happened.”

The change: bumping an image tag from v1.4.1 to v1.4.2 in apps/checkout/deployment.yaml. Auto-sync with self-heal is on. Argo CD manages the cluster it runs in.

Step 1 — the commit lands

git commit -am "checkout v1.4.2" && git push

Nothing happens in Argo CD. Git does not notify anyone; there is no push subscription. Something has to ask.

Step 2 — Argo CD notices, one of two ways

Polling. The application controller re-checks every Application’s repository every 3 minutes by default (timeout.reconciliation). It runs git ls-remote to resolve targetRevision to a SHA and compares it with the one it last rendered.

Webhook. If you configured one, the git host POSTs to /api/webhook on the API server the moment you push, and the matching Applications are refreshed immediately.

   no webhook:   0 to 180 seconds of doing nothing
   webhook:      under a second

This step is the entire answer to “why is Argo CD slow?” in the large majority of cases. It is not slow. It has not looked yet.

Configure the webhook. It is one URL and a shared secret, and it changes deployment from “up to three minutes” to “immediate”.

Worth knowing: git ls-remote is cheap but not free. A thousand Applications polling every three minutes is a request to your git host every 0.18 seconds, forever. On GitHub that meets rate limits. Webhooks fix the load problem as well as the latency one, and chapter 13 covers raising the interval once you have them.

Step 3 — the repo server renders

The controller asks the repo server for the manifests at the new SHA.

   repo server:
     1. is this (repo, SHA, path, params) in Redis?
          hit  → return it, ~5 ms
          miss → continue
     2. fetch the repository (cached clone, so usually
        an incremental fetch, not a full clone)
     3. check out the SHA
     4. detect: Chart.yaml? kustomization.yaml? plain?
     5. run the renderer
     6. cache the result in Redis
     7. return a flat list of objects

Cost: 5 ms cached; 1–30 seconds cold, dominated by the fetch and the renderer. A large Helm chart with many dependencies, or a repository with a long history and no shallow clone, is where the seconds go.

A subtlety that produces confusing behaviour: the repo server holds a per-repository lock while fetching. Fifty Applications from the same repository, all refreshing after one push, queue behind one fetch. That is usually fine and occasionally the reason a big monorepo takes minutes to propagate — chapter 13 covers reposerver.parallelism.limit and why more replicas do not always help.

Step 4 — the controller reads live state

No API calls here. The controller already holds informer caches for every managed cluster, so live state is read from memory.

   for each desired object:
     look up the live object in the cluster cache
       → found:     compare
       → not found: it needs creating
   for each live object tracked by this Application:
     not in desired? → it needs pruning

That second half is why deleting a file from git can result in a deletion: Argo CD knows which live resources carry its tracking annotation (chapter 3) and notices that one is no longer desired.

Cost: milliseconds. Reads from memory.

Step 5 — the diff

Argo CD normalises both sides and compares them. This is the most intricate step in the whole system and chapter 6 is dedicated to it. In outline:

   1. normalise desired (drop nulls, apply known defaults)
   2. normalise live (strip status, managedFields,
      server-populated fields, and anything in
      ignoreDifferences)
   3. perform a three-way merge against last-applied
      to decide which differences MATTER
   4. any real difference → OutOfSync

Here, image: acme/checkout:v1.4.1 versus v1.4.2 is one changed field in one object.

   Sync Status: OutOfSync
   Diff:        1 resource, Deployment/checkout

Cost: milliseconds to seconds, depending on object count.

Step 6 — the sync decision

Auto-sync is on, so the controller decides to act. It checks, in order:

   ✓ is auto-sync enabled?
   ✓ is a sync window open?          (chapter 10)
   ✓ has this exact revision already failed?
       → if yes, respect the retry backoff
   ✓ does the AppProject permit these resources
     in this destination?
   → proceed

That third check is worth knowing: Argo CD will not retry the same failing revision in a tight loop. It backs off. A sync that fails on a bad manifest does not hammer your cluster; it waits, which is why a broken commit shows as failed for a while rather than as a storm.

Step 7 — apply

The controller groups resources into sync waves (chapter 7) and applies them wave by wave.

   wave -1: namespaces, CRDs           (none changed here)
   wave  0: everything else            ← our Deployment
   wave  1: post-install jobs          (none)

Within a wave, resources are applied in a fixed order — namespaces, then CRDs, then RBAC, then everything else — so dependencies work out. Then the equivalent of:

kubectl apply -f - --server-side   # if ServerSideApply=true

With ApplyOutOfSyncOnly=true, only the changed Deployment is sent. Without it, every object in the Application is applied every sync — correct, idempotent, and much slower on a large app.

Cost: 100 ms to several seconds, mostly the Kubernetes API server’s admission chain.

At this moment the Application flips to Synced. This is the point people misread. Synced means “the cluster’s spec now matches git”. The new pod has not started. Nothing has been verified. Which is why:

Step 8 — health assessment

The controller now watches the resources it touched and assesses health (chapter 8).

   Deployment/checkout
     → Progressing:  new ReplicaSet created,
                     0/5 updated replicas ready
     → Progressing:  3/5 ready
     → Healthy:      5/5 ready, observedGeneration
                     matches, no unavailable replicas

Health comes from Kubernetes’ own status fields, watched through the informer, so it updates as the rollout progresses.

Cost: as long as the rollout takes — seconds to minutes, and it is Kubernetes’ work, not Argo CD’s.

If the new image does not exist, the Deployment never becomes ready, and after progressDeadlineSeconds (600 by default) Kubernetes marks it failed:

   Sync Status:   Synced
   Health Status: Degraded

Synced and Degraded together is the signature of “Argo CD did exactly what git said, and git said something broken.” It is not an Argo CD failure and no amount of re-syncing will fix it. Fix the manifest.

The whole picture

   git push
     │
     ├─ webhook ────────────────► under 1 s
     └─ or poll ────────────────► up to 180 s   ◄── usually the delay
     │
     ▼
   repo server renders            5 ms cached / 1–30 s cold
     │
     ▼
   controller reads live cache    ~0 ms
     │
     ▼
   diff                           ms–s
     │
     ▼
   apply (waves, hooks)           100 ms–s     → Synced
     │
     ▼
   health assessment              rollout time → Healthy

Total with a webhook: a few seconds plus your rollout time. Without: add up to three minutes of nothing.

Self-heal: the same loop from the other end

Now someone runs kubectl scale deployment checkout --replicas=10.

   t=0     the scale command succeeds
   t=0.1s  the controller's informer fires — live state changed
   t=0.1s  diff: replicas 10 (live) vs 5 (git) → OutOfSync
   t=?     self-heal triggers a sync
   t=?+1s  replicas back to 5

Note the ?. Self-heal is not instant, deliberately. There is a debounce (selfHealTimeout, 5 seconds by default) and a backoff on repeated healing, so that a controller writing to a field does not produce an infinite fight.

That debounce is exactly what makes ignoreDifferences necessary for HPA-managed replicas (chapter 3). Without the ignore, the HPA scales up, Argo CD scales down, the HPA scales up again — and the debounce turns an infinite loop into a slow infinite loop rather than preventing it.

Where the delay actually is

The triage table:

Symptom	Step	Fix
nothing happens for ~3 minutes	2	configure a webhook
nothing happens ever	2	wrong targetRevision, or the webhook is not firing
shows an old commit after a push	3	moved mutable tag, or submodule — --hard-refresh
refresh takes 30+ seconds	3	slow render; big chart, deep history, slow plugin
Synced immediately, nothing running	8	it applied; the rollout is Kubernetes’ problem
Synced but Degraded	8	git describes something broken
flapping OutOfSync ↔ Synced	5	another controller owns a field — ignoreDifferences
sync fails and retries slowly	6	working as designed — read the error

What to take from this chapter

• Git notifies nobody. Argo CD finds out by polling every 3 minutes or by webhook. Configure the webhook — this is the answer to almost every “Argo CD is slow” complaint, and it fixes git-host rate limiting too.
• Rendering is cached by repo + SHA + parameters; cold renders cost seconds and hold a per-repository lock, so a monorepo’s Applications queue behind one fetch.
• Live state comes from in-memory informer caches, not API calls — which is why drift is noticed in milliseconds.
• The diff is where the intricacy is (chapter 6); the apply is grouped into waves and can be limited to changed resources with ApplyOutOfSyncOnly=true.
• Synced happens at apply time, before anything is verified. Health is assessed afterwards, and takes as long as the rollout takes.
• Synced + Degraded means git describes something broken. Re-syncing will not help.
• A failing revision backs off rather than retrying in a loop.
• Self-heal is debounced (5 s) and backs off, which is why an HPA fighting Argo CD produces a slow loop rather than a fast one — the real fix is ignoreDifferences.

Next: diffing — the three-way merge, why things show OutOfSync forever, and when server-side apply is the answer.