6. Diffing

The diff is where Argo CD is cleverest and where it confuses people most. “Why does this say OutOfSync when nothing has changed?” is the most-asked Argo CD question in existence, and every answer is in this chapter.

The problem with a naive diff

You might expect Argo CD to compare git’s YAML with the cluster’s YAML and call any difference drift. That would report everything as permanently out of sync, because Kubernetes changes every object you give it.

# what you wrote
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: acme/checkout:v1.4.2

# what the cluster has, moments later
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop                        # defaulted
  uid: 8f3c...                           # assigned
  generation: 4                          # assigned
  creationTimestamp: "2026-01-14T…"      # assigned
  resourceVersion: "88213"               # assigned
  managedFields: [ … ]                   # server-side apply bookkeeping
  annotations:
    deployment.kubernetes.io/revision: "3"
spec:
  replicas: 3
  revisionHistoryLimit: 10               # defaulted
  strategy:                              # defaulted
    type: RollingUpdate
    rollingUpdate: { maxSurge: 25%, maxUnavailable: 25% }
  template:
    spec:
      dnsPolicy: ClusterFirst            # defaulted
      restartPolicy: Always              # defaulted
      terminationGracePeriodSeconds: 30  # defaulted
      containers:
        - name: app
          image: acme/checkout:v1.4.2
          imagePullPolicy: IfNotPresent  # defaulted
          terminationMessagePath: …      # defaulted
status: { … }                            # entirely server-owned

Dozens of differences, none of them drift. Every one is Kubernetes doing its job. Argo CD has to tell “you changed the image” apart from “the API server filled in a default”, and that is what the rest of this chapter is about.

Normalisation

Before comparing, Argo CD strips both sides of things that cannot be drift:

Always removed: status, metadata.creationTimestamp, metadata.resourceVersion, metadata.uid, metadata.generation, metadata.managedFields, metadata.selfLink.

Built-in normalisers handle known-awkward cases: CPU quantities (1 vs 1000m vs 1.0), memory (1Gi vs 1073741824), empty-versus-absent collections, and a set of well-known controller-owned fields.

Your ignoreDifferences are removed too.

System-level normalisation in argocd-cm applies across every Application:

data:
  resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
    jsonPointers:
      - /webhooks/0/clientConfig/caBundle
  resource.customizations.ignoreDifferences.all: |
    managedFieldsManagers:
      - kube-controller-manager

That caBundle example is the canonical one: cert-manager injects a CA bundle into webhook configurations after they are created, and without ignoring it every webhook is permanently out of sync.

The three-way merge

Even after normalisation, defaults remain. Git says nothing about imagePullPolicy; the cluster says IfNotPresent. Is that drift?

No — and working out why requires a third input.

   DESIRED       what git says now
   LIVE          what the cluster has
   LAST-APPLIED  what Argo CD sent last time

The rule:

   in LAST-APPLIED and in DESIRED?
     → compare them. Different = drift.

   in LIVE but NOT in LAST-APPLIED?
     → we never set it. Somebody or something else
       owns it. NOT drift.

   in LAST-APPLIED but NOT in DESIRED?
     → we used to set it and no longer do.
       It must be REMOVED. This IS drift.

That middle rule is the whole trick. imagePullPolicy is in live and not in last-applied, so Argo CD ignores it. Change replicas from 3 to 5 in git and it is in both, differs, and is drift.

The third rule is what makes removing a field work. Delete nodeSelector from your manifest and Argo CD notices it is in last-applied but no longer desired, and removes it — where a naive “apply what git says” would leave it in place forever.

last-applied is the kubectl.kubernetes.io/last-applied-configuration annotation, which is exactly the same mechanism kubectl apply uses. Argo CD’s client-side apply is kubectl apply’s algorithm.

The annotation limit, and server-side apply

That annotation is a real Kubernetes annotation, and annotations are capped at 262,144 bytes in total.

   Error: metadata.annotations: Too long: must have at most
   262144 bytes

This is the classic failure with large CRDs. Prometheus Operator’s, cert-manager’s, and Istio’s CRDs are hundreds of kilobytes of OpenAPI schema, and storing a copy of each in its own annotation exceeds the limit.

Server-side apply is the fix, and it is a better mechanism generally:

syncPolicy:
  syncOptions:
    - ServerSideApply=true

With server-side apply there is no last-applied annotation. The API server itself tracks field ownership in metadata.managedFields: each field records which manager set it. Argo CD declares itself a manager, sets its fields, and the server works out the merge — including telling Argo CD when it is trying to take a field another controller owns.

	Client-side apply	Server-side apply
Where the merge happens	in Argo CD	in the API server
Previous state stored as	an annotation (256 KB cap)	`managedFields`
Large CRDs	fails	works
Field ownership	not tracked	tracked per manager
Conflicts with other controllers	silent	explicit

Turn it on. It is the direction Kubernetes has gone, it removes an entire class of failure, and it makes ignoreDifferences with managedFieldsManagers possible — the cleanest way to say “the HPA owns replicas, leave them alone” without naming a JSON pointer.

One caveat: switching an existing Application from client-side to server-side apply causes one large resync as ownership is established, and can surface conflicts that were previously silent. Do it on a quiet day, not during an incident.

Why it says OutOfSync forever

The catalogue, in order of how often each is the answer.

1. Another controller owns a field. An HPA writes spec.replicas; a mutating webhook injects a sidecar; a service mesh adds annotations. Git says one thing, the controller writes another, and with self-heal on they fight.

Fix: ignoreDifferences, preferably with managedFieldsManagers.

2. A defaulted field you also specified. You wrote protocol: TCP on a Service port; the API server normalises something adjacent; the shapes differ. Or an empty map versus an absent one.

Fix: a normaliser, or stop specifying the default.

3. null versus absent. Helm templates that render key: null produce a field Kubernetes strips. The diff shows it as removed forever.

Fix: do not render null keys; guard with {{- if }}.

4. Number formatting. 1 vs 1.0 vs "1". A Helm --set turning 1.10 into 1.1.

Fix: quote strings, use stringParameters.

5. A mutating admission webhook. Istio’s sidecar injection, Vault’s agent injector, Linkerd’s proxy. The live pod template has containers git never mentioned.

Fix: ignoreDifferences on the injected paths, or exclude the resource. This one is genuinely annoying and there is no elegant answer.

6. The resource is genuinely different. Someone changed it. That is the tool working.

Reading a diff

argocd app diff checkout
argocd app diff checkout --local ./apps/checkout   # against local files
argocd app diff checkout --server-side-generate    # render server-side

The UI’s diff view has two toggles worth knowing:

“Compact diff” hides unchanged lines. “Include ignored” shows differences that ignoreDifferences is hiding — which is how you check that an ignore rule is doing what you think, rather than quietly masking a real change.

That second one deserves a habit: periodically look at what your ignores are hiding. An ignoreDifferences entry added a year ago to silence an HPA can end up concealing that someone hand-edited a resource limit.

Diff customisations with Lua

For the awkward cases, argocd-cm accepts Lua that decides whether a difference matters:

data:
  resource.customizations.knownTypeFields.argoproj.io_Rollout: |
    - field: spec.template.spec
      type: core/v1/PodSpec

knownTypeFields tells Argo CD that a field inside a custom resource has a known Kubernetes type, so the built-in normalisers — CPU and memory quantities in particular — apply inside it. Without this, an Argo Rollouts Rollout with cpu: 1 in git and cpu: 1000m live shows as permanently out of sync, because Argo CD has no way to know that field is a PodSpec.

When to ignore, and when not to

The line worth holding:

   IGNORE when another system legitimately OWNS the field
     ✓ HPA owning spec.replicas
     ✓ cert-manager injecting caBundle
     ✓ a mesh injecting sidecars
     ✓ a cloud controller writing a load balancer IP

   DO NOT IGNORE to silence something you have not
   understood
     ✗ "the diff is noisy, ignore the whole spec"
     ✗ ignoring status-like fields on a CRD instead of
       writing a health check
     ✗ ignoring a field because a human keeps editing it
       — that is drift, and it is the thing you installed
         Argo CD to detect

Every ignore is a hole in the guarantee that git describes the cluster. A few precise holes are fine and necessary. A broad one — jsonPointers: ["/spec"] — means the Application is decorative.

What to take from this chapter

A naive diff is impossible: Kubernetes defaults and populates dozens of fields on every object.
Argo CD normalises both sides (drops status, managedFields, timestamps; handles CPU/memory quantities) before comparing.
The three-way merge uses desired, live, and last-applied: a field in live but not last-applied is somebody else’s and is ignored; a field in last-applied but no longer desired is removed.
Client-side apply stores last-applied in an annotation, and annotations are capped at 256 KB — which is why large CRDs fail.
Turn on ServerSideApply=true. The API server does the merge and tracks field ownership, removing that failure class and making conflicts explicit.
“OutOfSync forever” is almost always: another controller owns a field, a defaulted field, null vs absent, number formatting, or a mutating webhook.
Prefer managedFieldsManagers over JSON pointers in ignoreDifferences — name the owner, not the field.
Use knownTypeFields so quantity normalisation works inside custom resources.
Check what your ignores are hiding, periodically. Ignore what another system legitimately owns; never ignore to silence something you have not understood.

Next: sync — waves, hooks, prune, and the ordering machinery that makes complicated deployments work.

Written by Sepuri Sai Krishna