14. Failure modes

A catalogue. Each entry: what you see, what is happening, what to do, how to prevent it.

The pattern worth noticing: most Argo CD incidents are Argo CD correctly doing what it was told. The tool is unusually literal, and the failures are almost all a gap between what someone meant and what git said.

1. The sync that never finishes

Symptoms. The Application sits in Progressing or Syncing forever. Some resources are applied; later ones are not. The UI shows a spinner and no error.

Cause. A sync wave is blocked. Argo CD waits for every resource in a wave to be Healthy before starting the next, and something in an early wave never gets there.

Usual culprits: a PreSync Job that fails or hangs; a custom resource whose operator is not running; a Deployment with a bad image; a PVC that cannot bind because no storage class matches.

What to do.

argocd app get checkout --show-operation     # which resource, which wave
argocd app terminate-op checkout             # cancel the stuck sync
kubectl describe <the blocked resource>      # the real error

The --show-operation output names the phase and the resource. Nine times out of ten kubectl describe on that resource has the answer immediately.

Prevention. Fewer waves. Set progressDeadlineSeconds on Deployments so a failed rollout fails rather than hanging. Give hook Jobs activeDeadlineSeconds and backoffLimit: 0. And write custom health checks for custom resources (chapter 8) — without them a wave passes the moment a resource exists, which is a different bug in the same machinery.

2. Permanently OutOfSync

Symptoms. OutOfSync immediately after a successful sync, forever. The diff shows a field you did not set, or one you did.

Cause. Chapter 6’s list: another controller owns the field, a defaulted value, null vs absent, number formatting, or a mutating webhook.

What to do.

argocd app diff checkout

Then decide which it is. If another controller legitimately owns it, ignoreDifferences — preferably with managedFieldsManagers rather than a JSON pointer.

Prevention. Turn on ServerSideApply=true, which makes ownership explicit rather than something you have to infer from a diff.

3. Self-heal fighting a controller

Symptoms. A resource flips between two states every few minutes. The Application alternates Synced and OutOfSync. Sync history is full of identical syncs.

Cause. Two controllers want different values. Classically an HPA writes spec.replicas: 12; git says 3; self-heal writes 3; the HPA writes 12.

What to do. ignoreDifferences on /spec/replicas for that Deployment. And remove replicas from the manifest entirely — if an HPA owns it, git should not state it.

Prevention. When adopting an HPA, do both at once: delete replicas from the manifest and add the ignore. Half of this does not work.

4. Self-heal fighting a human during an incident

Symptoms. An engineer scales up a deployment during an outage. Minutes later it scales back down. They do it again. It reverts again.

Cause. Self-heal is doing exactly its job. Nobody told the person that git is the only way to change the cluster, or they knew and forgot at 3 a.m.

What to do — in the moment:

argocd app set checkout --sync-policy none    # stop auto-sync, keep the app
# fix the incident
# then commit the fix and re-enable

Or argocd app patch to add the change to ignoreDifferences temporarily. Both are better than deleting the Application, which is what people reach for under pressure and which is how an incident becomes an outage.

Prevention. Make the emergency procedure written down and practised. Everyone who is ever on call should know the “how do I stop Argo CD reverting me” command before they need it. This is a real operational cost of self-heal and it deserves a runbook line, not a discovery.

5. Argo CD cannot fix itself

Symptoms. A commit to Argo CD’s own manifests breaks Argo CD. It is now unable to sync the commit that would fix it.

Cause. Argo CD managing itself is a genuine circular dependency. A bad values change to the Argo CD Helm chart — a wrong image tag, a broken RBAC config, an OOMing controller — and the thing that would apply the correction is the thing that is broken.

What to do. Break the circle by hand.

git revert <the bad commit> && git push
# then apply it directly, bypassing Argo CD:
helm template argocd argo/argo-cd -f values.yaml | kubectl apply -f -
# or:
kubectl apply -k <the argocd overlay>

Prevention.

Manage Argo CD manually or from a separate instance, not from itself, if you can bear the inconsistency. A small management cluster whose Argo CD manages the workload clusters — and whose own Argo CD is installed by Terraform — removes the circularity entirely.

If Argo CD does manage itself: use manual sync for that one Application, so a bad commit does not deploy itself unattended.

And know the recovery command before you need it. This is the failure where knowing the answer in advance is worth the most, because it is not obvious under pressure that kubectl apply is allowed.

6. Deleting an Application deleted production

Symptoms. kubectl delete application checkout — and every workload is gone.

Cause. The resources-finalizer.argocd.argoproj.io finalizer cascades deletion to everything the Application created. That is what it is for.

What to do. Re-sync from git. Stateless workloads come back in minutes. PersistentVolumeClaims may not — depending on the reclaim policy, the underlying volume may already be gone. This is the part that turns an embarrassment into an incident.

Prevention.

# on anything holding data
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false

Set the StorageClass reclaimPolicy: Retain for anything you care about. Restrict applications, delete in Argo CD RBAC (chapter 10) to very few people. And use the UI’s cascade toggle deliberately — argocd app delete --cascade=false removes the Application and leaves the workloads.

7. An ApplicationSet deleted everything

Symptoms. A batch of Applications vanish at once, with their workloads.

Cause. Chapter 11’s behaviour. Either the ApplicationSet itself was deleted, or its generator returned an empty or smaller set — a deleted git branch, a cluster label removed, a repository made private, a git generator pointed at a path that no longer exists.

The branch case is the nastiest: a CI job that prunes merged branches deletes the branch a git generator scans, the generator returns nothing, and every generated Application is deleted.

What to do. Restore the generator’s input, and the Applications regenerate. Workloads re-sync. PVCs, again, may not survive.

Prevention.

spec:
  syncPolicy:
    applicationsSync: create-update      # never delete automatically
    preserveResourcesOnDeletion: true

Set create-update in production. Removing an application should be a deliberate act, not a side effect of a generator returning fewer results.

8. Repo server out of memory

Symptoms. Repo server pods OOMKilled. Renders time out. Everything is stuck at “Refreshing”.

Cause. Too many concurrent renders of large charts, or one enormous repository, or a config management plugin holding a lot in memory. A push to a monorepo triggering four hundred simultaneous renders will do it.

What to do. Raise the memory limit; set reposerver.parallelism.limit to bound concurrency.

Prevention. Bound parallelism, use shallow clones, split genuinely huge monorepos, and keep an eye on plugin memory. More replicas do not help if each one is trying to do everything at once.

9. Git host rate limiting

Symptoms. Applications stop updating. Repo server logs show 403s or rate limit exceeded. Nothing appears broken.

Cause. No webhooks, so every Application runs git ls-remote every three minutes. A thousand Applications is 20,000 requests an hour.

What to do. Configure webhooks; raise timeout.reconciliation; switch from personal access tokens to a GitHub App, which has substantially higher limits.

Prevention. All three, from the start.

10. Redis down

Symptoms. The UI times out. Reconciliation is very slow. Nothing is obviously broken and nothing is obviously working.

Cause. Cache misses everywhere. Argo CD treats them as expensive, not fatal, so it degrades rather than failing — which is harder to diagnose than a clean failure.

What to do. Restart Redis. Nothing is lost; it is a cache (chapter 2).

Prevention. Run the HA manifests. The default single-replica Redis is the least redundant part of the default install and the one whose loss is most visible.

11. A sync applied to the wrong cluster

Symptoms. Resources appear in an environment nobody intended.

Cause. A destination typo, or an ApplicationSet cluster generator with a selector matching more clusters than expected — someone labelled a production cluster env: dev by mistake, or a matchLabels was loosened.

What to do. Delete the Application with cascade, carefully, having checked what it created.

Prevention. This is what AppProject destinations are for. A project restricted to specific cluster/namespace pairs makes the wrong destination a rejected Application rather than a deployment. If you take one thing from chapter 10, it is that projects turn a class of human error into a validation failure.

12. Diagnosis order

When something is wrong and you do not know where to start:

   1. argocd app get <app>
      → sync status, health, last operation, error message

   2. argocd app get <app> --show-operation
      → which phase, which resource, what failed

   3. argocd app diff <app>
      → what does Argo CD think is different?

   4. argocd app history <app>
      → what changed, and when did it last work?

   5. kubectl describe <the failing resource>
      → the actual Kubernetes error

   6. kubectl logs -n argocd deploy/argocd-repo-server
      kubectl logs -n argocd statefulset/argocd-application-controller
      → render failures live in the first;
        reconciliation problems in the second

Steps 1–4 answer most questions without leaving the argocd CLI. Step 5 is where Kubernetes errors — image pull failures, quota, admission webhooks — surface, and those are the majority of “Argo CD is broken” reports.

What to take from this chapter

Most Argo CD incidents are Argo CD doing exactly what it was told. The gap is between what someone meant and what git said.
Stuck syncs are blocked waves. Use --show-operation, then kubectl describe. Fewer waves, deadlines on hooks, health checks for custom resources.
Permanent OutOfSync is chapter 6’s list — fix with ignoreDifferences (prefer managedFieldsManagers) and ServerSideApply=true.
Self-heal versus an HPA: ignore /spec/replicas and remove it from the manifest. Half of this does not work.
Self-heal versus a human in an incident: argocd app set <app> --sync-policy none. Put this in the runbook before you need it.
Argo CD managing itself cannot fix itself. Recover with kubectl apply by hand; prefer managing it from elsewhere, or at least with manual sync.
Deleting an Application deletes its workloads. Protect data with Prune=false,Delete=false, reclaimPolicy: Retain, and RBAC on applications, delete.
ApplicationSets delete in bulk when a generator returns less — a pruned git branch is enough. applicationsSync: create-update in production.
Repo server OOM: bound reposerver.parallelism.limit. Git rate limiting: webhooks plus a GitHub App. Redis down: restart it, run HA.
AppProject destinations turn “deployed to the wrong cluster” into a rejected Application.

Next: when to use Argo CD and when not — the comparison with Flux, with pushing from CI, and with Terraform.

Written by Sepuri Sai Krishna