8. Health

Sync status answers “does the cluster match git?”. Health answers “is it working?”. They are computed differently, from different inputs, and they are independent — a fact worth repeating because it is the source of most confusion about what Argo CD is telling you.

The five states

State	Meaning	Colour in the UI
Healthy	working as intended	green
Progressing	not there yet, but not failed	blue, spinning
Degraded	failed	red
Suspended	intentionally paused (a suspended CronJob, a paused Rollout)	grey
Missing	in git, not in the cluster	grey
Unknown	no health check exists for this kind	grey

Progressing and Degraded are the two that carry information. Progressing for thirty seconds is a rollout; Progressing for thirty minutes is a rollout that is never going to finish and whose deadline has not yet expired.

How the built-in checks work

Health is derived from the resource’s own status fields, which Kubernetes controllers already maintain. Argo CD is reading, not probing — it never opens a connection to your application.

Deployment

   Healthy if:
     status.observedGeneration >= metadata.generation
     AND status.updatedReplicas == spec.replicas
     AND status.availableReplicas == spec.replicas
   Degraded if:
     a condition Progressing=False with reason
     ProgressDeadlineExceeded
   otherwise Progressing

The observedGeneration check is the important one. It means “the controller has seen my latest change” — without it, an unchanged old ReplicaSet would look healthy immediately after you applied a new spec.

StatefulSet — same idea, plus partitioned rollout awareness.

Pod — Running with all containers ready is healthy; Succeeded is healthy; Failed is degraded; CrashLoopBackOff is degraded.

Job — a Complete condition is healthy; Failed is degraded; otherwise progressing.

Service — healthy immediately, except type: LoadBalancer, which is progressing until status.loadBalancer.ingress is populated. This is genuinely useful: an Application stays Progressing until the cloud actually gave you an address.

Ingress — same, waiting for status.loadBalancer.

PersistentVolumeClaim — Bound is healthy; Pending is progressing.

Argo Rollouts Rollout — has a built-in check, which is why Rollouts and Argo CD compose so well.

Anything else — Healthy if it exists, unless a custom check says otherwise. This is the important default: a custom resource with no health check is considered healthy the moment it exists, regardless of what its controller thinks.

The aggregate

An Application’s health is the worst health of any resource it manages, with a precedence order:

   Degraded  >  Progressing  >  Healthy  >  Suspended  >  Missing

So one crash-looping pod out of two hundred resources makes the whole Application Degraded. That is usually right, and it produces one predictable annoyance: a CronJob whose last run failed leaves the Application red even though nothing is actually wrong with the service.

The escape:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

or a custom health check for that kind that is less strict. Use them sparingly — an Application that is permanently amber teaches everyone to ignore the colour, which is worse than the original problem.

Custom health checks

For custom resources, a Lua script in argocd-cm:

data:
  resource.customizations.health.example.com_Database: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.phase == "Ready" then
        hs.status = "Healthy"
        hs.message = "Database is ready"
        return hs
      end
      if obj.status.phase == "Failed" then
        hs.status = "Degraded"
        hs.message = obj.status.message or "Database failed"
        return hs
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for database"
    return hs

The contract: you get obj — the live resource as a Lua table — and you return a table with status and message.

Three rules learned the hard way:

Always guard for nil. A resource that has just been created has no status at all, and indexing nil throws — which Argo CD reports as Unknown with a Lua error, not as the health state you intended.

Never default to Healthy. Return Progressing when you cannot tell. A check that returns Healthy for an unknown state is worse than no check, because it makes sync waves proceed past something that is not ready.

Write a useful message. It is what appears in the UI and in argocd app get, and it is the difference between “Degraded” and “Degraded: image pull backoff on container app”.

Test before deploying:

argocd admin settings resource-overrides health ./database.yaml \
  --argocd-cm-path ./argocd-cm.yaml

This is worth doing every time. A Lua error in a health check is not obvious from the UI.

Why this matters more than it looks

Custom health checks are what make sync waves work with operators.

   wave 0: apply a `Database` custom resource
   wave 1: apply the Deployment that connects to it

   with no health check:
     the Database is "healthy" the instant it exists
     → wave 1 runs immediately
     → the app starts before the database exists
     → crash loop

   with a health check:
     wave 0 waits for status.phase == "Ready"
     → wave 1 runs when the database is genuinely up

If you use operators and sync waves together, you need custom health checks. Without them the waves are decorative. Many popular operators — cert-manager, Crossplane, the Prometheus operator, most database operators — ship checks in Argo CD’s bundled library, so check before writing your own.

The combinations, and what each means

The table worth internalising:

Sync	Health	Meaning	Action
Synced	Healthy	everything is right	none
Synced	Progressing	applied, rolling out	wait
Synced	Degraded	git describes something broken	fix the manifest — re-syncing will not help
OutOfSync	Healthy	running, but not what git says	sync, or investigate drift
OutOfSync	Degraded	broken and wrong	look at both
Synced	Missing	applied but the resource is gone	something deleted it; check for a competing controller
Unknown	any	a health check errored	check the Lua

The third row is the one people misread most often. Synced + Degraded is not an Argo CD problem. Argo CD applied precisely what git contained; the result does not work. Clicking Sync again applies the same broken thing.

Watching health from outside

argocd app get checkout                       # summary
argocd app get checkout --show-operation      # last sync's detail
argocd app resources checkout                 # per-resource health
argocd app logs checkout --follow             # logs of managed pods
argocd app wait checkout --health --timeout 600

The Prometheus metrics are the ones to alert on:

   argocd_app_info{health_status="Degraded"}
   argocd_app_info{sync_status="OutOfSync"}

Both are gauges labelled per Application, so:

   # an application has been unhealthy for 15 minutes
   argocd_app_info{health_status="Degraded"} == 1

   # drifted and not being corrected for an hour
   argocd_app_info{sync_status="OutOfSync"} == 1

Alert on duration, not on the instantaneous value. Every deployment passes through Progressing and briefly through OutOfSync; only a state that persists is a problem. If you have read the Prometheus alerting chapter, this is the standard for: clause argument, and the right values are roughly 15 minutes for Degraded and an hour for OutOfSync.

What health does not tell you

Worth being explicit, because it is easy to over-trust the green tick.

It does not mean your application works. It means Kubernetes’ controllers are satisfied. A Deployment whose pods pass their readiness probe but return 500 to every real request is Healthy.

It does not check dependencies. An application healthy in every Kubernetes sense, that cannot reach its database, is Healthy.

It does not test anything. There is no request made, no endpoint hit. If you want that, it is a PostSync hook running an actual smoke test — which is worth doing, and which is the only thing in this chapter that verifies the application rather than the manifest.

Argo CD tells you the cluster matches git and that Kubernetes is content. Whether the software works is still your monitoring’s job, and conflating the two is how you get a green dashboard during an outage.

What to take from this chapter

• Five states: Healthy, Progressing, Degraded, Suspended, Missing (plus Unknown). Health is read from the resource’s own status fields — Argo CD never probes your application.
• Deployment health requires observedGeneration caught up plus updated and available replicas matching — which is what stops a stale ReplicaSet looking healthy.
• A custom resource with no health check is Healthy the moment it exists. This is the default that breaks sync waves with operators.
• An Application’s health is the worst of its resources: Degraded > Progressing > Healthy > Suspended > Missing.
• Custom checks are Lua in argocd-cm. Always guard for nil status, never default to Healthy, and write a useful message. Test with argocd admin settings resource-overrides health.
• Operators plus sync waves require custom health checks, or the waves do nothing. Many popular operators are already in the bundled library.
• Synced + Degraded means git describes something broken. Re-syncing applies the same broken thing.
• Alert on argocd_app_info duration, not instantaneous state — every deployment passes through Progressing.
• Healthy does not mean working. It means Kubernetes is satisfied. Verifying the application is a PostSync smoke test and your monitoring, not this.

Next: repositories and credentials — private repos, SSH, GitHub Apps, and the credential templates that stop you configuring fifty repositories one at a time.