7. Sync

Sync is the act of making live match desired. For a simple application it is one kubectl apply and nothing in this chapter matters. For anything with ordering requirements — CRDs before the resources that use them, a migration before the new code, a namespace before what goes in it — this chapter is the whole game.

The default order

Even with no configuration, Argo CD does not apply resources in a random order. Within a wave it uses a fixed sequence, derived from what usually depends on what:

   Namespace
   ResourceQuota, LimitRange
   PodSecurityPolicy / policy resources
   CustomResourceDefinition
   ServiceAccount, Secret, ConfigMap
   ClusterRole, ClusterRoleBinding, Role, RoleBinding
   Service, Endpoints
   PersistentVolume, PersistentVolumeClaim
   DaemonSet, Deployment, StatefulSet, ReplicaSet, Pod
   Job, CronJob
   Ingress
   everything else, including custom resources

This handles most cases for free. A Deployment referencing a ConfigMap works because ConfigMaps come first. CRDs come before the custom resources in the same wave.

Where it breaks down is anything the list cannot know about: your database must be migrated before the new code starts; your operator must be running before its custom resources are created; an app must not start before a dependency in another Application is ready.

Sync waves

A wave is an integer annotation. All resources in a wave must be applied and healthy before the next wave starts.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Waves run in ascending order, and the default is 0 — so negative waves run before everything unannotated, which is exactly what you want for prerequisites.

   wave -2:  namespaces, CRDs
   wave -1:  operators, cert-manager, secrets
   wave  0:  ConfigMaps, Services, Deployments   ← the default
   wave  1:  Ingresses, custom resources
   wave  2:  smoke tests

The important part is the word healthy. Argo CD does not merely apply wave -1 and move on; it waits for every resource in it to be assessed Healthy (chapter 8) before starting wave 0. That is what makes waves useful and also what makes them dangerous:

A resource in an early wave that never becomes healthy blocks the entire sync forever.

A CRD whose controller is not running yet, a Job that fails, a Deployment with a bad image — the sync stops, and every later wave is never applied. Chapter 14 covers diagnosing this, because a “stuck” sync with a healthy-looking application is confusing.

Use as few waves as you can. Three or four for a complex application; not fifteen. Each one is a serialisation point, and a fifteen-wave application deploys slowly and fails opaquely.

Hooks

Hooks are resources that run at a point in the sync rather than being part of the desired state.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "0"
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: acme/checkout:v1.4.2
          command: ["./migrate", "up"]

The phases:

Hook	When	Typical use
PreSync	before any resource is applied	database migration
Sync	alongside the normal apply, in wave order	complex ordering
PostSync	after everything is applied and healthy	smoke test, cache warm, notification
SyncFail	only if the sync failed	alert, cleanup
Skip	never applied	resource present in git for reference only

A failing PreSync hook aborts the sync. Nothing is applied. That is the whole point: if the migration fails, the new code does not go out.

Delete policies

Hooks are real Kubernetes resources and they accumulate unless told otherwise:

Policy	Behaviour
HookSucceeded	delete after it succeeds — the usual choice
HookFailed	delete after it fails
BeforeHookCreation	delete the previous instance before creating a new one — the default

BeforeHookCreation being the default means the last run of each hook sticks around, which is useful for reading logs and is why you find old migration Jobs in your namespace. Combining BeforeHookCreation with HookSucceeded gives you: keep failures for debugging, clean up successes.

One real trap: a Job’s name must change between runs, or Kubernetes will reject the update — Job specs are largely immutable. Use generateName, or BeforeHookCreation so the old one is removed first.

Waves inside hooks

Hooks respect sync waves within their phase, so you can order several PreSync hooks:

   PreSync wave -1: wait for the database to be reachable
   PreSync wave  0: run the migration
   Sync:            apply everything
   PostSync wave 0: smoke test
   PostSync wave 1: notify Slack

Helm hooks

Helm’s chapter 9 covers what these are on the Helm side — including the fact that hook resources are not part of the release and are never rolled back, which is true here too.

Argo CD translates them:

Helm	Argo CD
pre-install, pre-upgrade	PreSync
post-install, post-upgrade	PostSync
post-delete	not supported
test	ignored

The mapping is good enough for most charts. post-delete having no equivalent occasionally matters for charts that clean up external resources on uninstall — you have to do that yourself.

The migration pattern

The question everyone eventually asks: how do you run a database migration safely?

   PreSync hook Job:
     image: THE NEW VERSION of your application
     command: migrate up
     restartPolicy: Never
     backoffLimit: 0        ← do not retry a half-applied
                              migration automatically

The rules that make it work:

Migrations must be backwards compatible with the currently running code. During the rollout both versions are live. Add a column, do not rename one. Two deployments — one that adds, one that removes — not one that renames.

Never destructive in a PreSync hook. Dropping a column before the old pods have stopped breaks them instantly and makes rollback impossible.

Idempotent. Hooks can be re-run — a retried sync runs them again.

Fail loudly. backoffLimit: 0 so a failure is a failure. A migration that half-applied and then retried is much worse than one that stopped.

If you have read the PostgreSQL locking chapter, the other half of this is there: a migration that takes an ACCESS EXCLUSIVE lock behind a long-running query queues every subsequent query and takes your application down, regardless of how correct the GitOps around it is. Set lock_timeout.

Prune

Deleting a manifest from git should delete the resource. With prune: false (the default) it does not — Argo CD reports it and leaves it alone.

syncPolicy:
  automated:
    prune: true
  syncOptions:
    - PruneLast=true

PruneLast=true deletes after everything else has synced and become healthy. Without it, deletions and creations are interleaved and you can get an unnecessary gap in service.

For resources that must never be deleted automatically:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

Put this on PersistentVolumeClaims and anything else holding data. A PVC removed from git because someone refactored a chart is a deleted volume, and the annotation is cheap insurance.

Also useful:

    argocd.argoproj.io/sync-options: Delete=false

which additionally protects the resource when the whole Application is deleted with the cascade finalizer.

Resources Argo CD should create but not manage

metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

for a custom resource whose CRD is created in the same sync — the dry-run validation would otherwise fail because the kind does not exist yet.

    argocd.argoproj.io/compare-options: IgnoreExtraneous

marks a resource as not counting towards sync status — useful for things another process creates that you do not want reported as out of sync.

Selective sync

argocd app sync checkout                      # everything
argocd app sync checkout --resource apps:Deployment:checkout
argocd app sync checkout --prune
argocd app sync checkout --dry-run
argocd app sync checkout --force              # kubectl replace
argocd app sync checkout --replace            # recreate resources

--force and --replace are the escape hatches for immutable-field errors — changing a Deployment’s selector, a Job’s spec, a Service’s clusterIP. They delete and recreate, which for a Deployment means downtime. They are correct answers to a real problem and they are not routine.

The other escape hatch:

argocd app terminate-op checkout   # cancel a stuck sync

Which you will need when a wave is blocked on something that will never become healthy.

Sync status of an in-progress operation

argocd app wait checkout --timeout 600
argocd app wait checkout --health --sync

argocd app wait is what you put in a pipeline that needs to know the deployment finished. It waits for both sync and health, which is the combination that actually means “it worked” — and is worth contrasting with the naive kubectl rollout status, which knows nothing about the other twelve resources in the Application.

What to take from this chapter

• Within a wave, Argo CD applies in a fixed kind order (namespaces → CRDs → RBAC → workloads → ingress), which handles most dependencies for free.
• Sync waves are integers; the default is 0, so negative waves run first. A wave completes only when every resource in it is Healthy.
• A resource that never becomes healthy blocks the sync forever. Use as few waves as possible.
• Hooks run at phases: PreSync (migrations — a failure aborts the sync), Sync, PostSync (after healthy), SyncFail, Skip.
• Hook delete policies default to BeforeHookCreation, so the last run sticks around. A hook Job’s name must change between runs or Kubernetes rejects it.
• Migrations: run the new version’s image in a PreSync hook, keep them backwards compatible and non-destructive, backoffLimit: 0, and set a database lock_timeout.
• prune is off by default; turn on PruneLast=true with it. Annotate PVCs with Prune=false and, if needed, Delete=false.
• --force / --replace fix immutable-field errors by recreating resources — which means downtime. Not routine.
• argocd app wait --health --sync is what a pipeline should block on; kubectl rollout status sees only one resource.

Next: health — how Argo CD decides something is working, custom Lua checks, and why “Synced” is not “working”.