13. Tuning and operations

Argo CD is undemanding until it isn’t. The transition happens somewhere between two hundred and a thousand Applications, and it arrives as reconciliation falling behind rather than as anything crashing.

This chapter is ordered by how much difference each change makes.

1. Configure webhooks

The single highest-value change, and it is five minutes of work.

   GitHub → repository settings → webhooks
     Payload URL:  https://argocd.acme.com/api/webhook
     Content type: application/json
     Secret:       (store as webhook.github.secret in argocd-secret)
     Events:       Just the push event

Without it, every Application polls its repository every three minutes. With it, the git host tells you.

Two benefits, and the second is the one people miss:

Latency goes from up to 180 seconds to under a second.

Load goes from N Applications × 20 requests per hour to one request per push. At a thousand Applications the polling version is 20,000 requests an hour to your git host, which meets rate limits and gets you throttled — at which point everything stops updating and the cause is not obvious.

Once webhooks work, raise the poll interval so it is a safety net rather than the mechanism:

# argocd-cm
data:
  timeout.reconciliation: 300s     # 5 minutes, or longer

2. Exclude resources you do not manage

The application controller builds an in-memory tree of every resource in every managed cluster, not just the ones Argo CD created. On a busy cluster most of that is noise, and it is the largest memory consumer in the system.

# argocd-cm
data:
  resource.exclusions: |
    - apiGroups: ["*"]
      kinds:
        - Event
        - EndpointSlice
        - Endpoints
        - PodMetrics
        - NodeMetrics
      clusters: ["*"]
    - apiGroups: ["cilium.io"]
      kinds: ["CiliumEndpoint", "CiliumIdentity"]
      clusters: ["*"]

Events are the big one. A busy cluster generates thousands, they churn constantly, and Argo CD does not need any of them. Excluding Events and EndpointSlices alone typically cuts controller memory by a large fraction and removes most of the watch traffic.

If you run a service mesh or a CNI that creates a per-pod custom resource — Cilium endpoints, Istio’s older resources — exclude those too. They change on every pod event and Argo CD has no use for them.

The inverse, resource.inclusions, is stricter and better when you know exactly what you manage — everything not listed is invisible. Safer on a large shared cluster, at the cost of having to remember to add new kinds.

3. Shard the application controller

The controller is the bottleneck, and it does not scale by adding replicas in the usual way. Each managed cluster is owned by exactly one controller shard.

# argocd-cmd-params-cm
data:
  controller.replicas: "3"
  controller.sharding.algorithm: "round-robin"

# the StatefulSet needs to know how many shards exist
env:
  - name: ARGOCD_CONTROLLER_REPLICAS
    value: "3"

Both settings are required and forgetting the second is a real trap: the controller reads the replica count from that variable, not from the StatefulSet, so a mismatch means clusters are assigned to shards that do not exist and their Applications are never reconciled — silently.

Sharding algorithms:

Algorithm	Behaviour
legacy	hash of cluster ID — uneven, the old default
round-robin	even distribution by index — usually the right choice
consistent-hashing	minimises reassignment when replica count changes

Sharding helps only if you manage multiple clusters. One cluster is one shard, and adding replicas achieves nothing. If you have a single large cluster and a slow controller, the fix is exclusions, ApplyOutOfSyncOnly, and vertical scaling — not shards.

Per-shard resources for a large installation:

resources:
  requests: { cpu: "1",  memory: "2Gi" }
  limits:   { memory: "8Gi" }     # no CPU limit

No CPU limit. The controller is bursty — a mass refresh after a push is a spike — and a CPU limit turns that spike into throttling, which turns into reconciliation falling behind. This is the same argument the Envoy tuning chapter makes about proxies, for the same reason.

4. Scale the repo server

Easy: it is stateless.

# argocd-cmd-params-cm
data:
  reposerver.parallelism.limit: "10"    # concurrent manifest generations

More replicas help when many different repositories are being rendered. They help less than you expect for one large monorepo, because the repo server holds a per-repository lock during fetch — fifty Applications from one repository queue behind one operation regardless of replica count.

For a monorepo the levers are: shallow clones, a faster renderer, and caching. reposerver.parallelism.limit bounds concurrent renders so a burst does not exhaust memory; set it if you see the repo server OOM after a large push.

5. Application-level settings that matter at scale

syncPolicy:
  syncOptions:
    - ApplyOutOfSyncOnly=true      # send only changed resources
    - ServerSideApply=true         # no 256 KB annotation limit

ApplyOutOfSyncOnly=true is the biggest per-Application win. Without it every sync applies every resource — correct, idempotent, and pointless work that scales with application size.

Notifications

Argo CD Notifications ships with it and is configured with triggers and templates:

# argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token

  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]

  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-degraded]

  template.app-degraded: |
    message: |
      {{.app.metadata.name}} is degraded.
      {{range .app.status.conditions}}{{.message}}
      {{end}}

Subscribe per Application:

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-health-degraded.slack: shop-alerts

Notify on failure, not on success. A channel that gets a message for every successful sync is a channel nobody reads, and the failure message arrives in a channel nobody reads. on-sync-failed and on-health-degraded are the two that earn their place.

Metrics and alerting

Argo CD exposes Prometheus metrics on port 8082 (controller), 8084 (repo server) and 8083 (API server).

# an Application unhealthy for 15 minutes
argocd_app_info{health_status="Degraded"} == 1
  for: 15m

# drifted and not corrected for an hour
argocd_app_info{sync_status="OutOfSync"} == 1
  for: 1h

# reconciliation falling behind — THE scaling signal
histogram_quantile(0.99,
  rate(argocd_app_reconcile_bucket[5m])) > 10

# sync failures
rate(argocd_app_sync_total{phase="Failed"}[15m]) > 0

# the controller cannot talk to a cluster
argocd_cluster_connection_status == 0

# git operations failing — usually rate limiting or credentials
rate(argocd_git_request_total{request_type="ls-remote"}[5m])

argocd_app_reconcile is the one to watch as you grow. It is the time to reconcile one Application, and when its p99 climbs past a few seconds you are approaching the point where shards or exclusions are needed. Alerting on it gives you warning before anyone notices deployments are slow.

Alert on duration, not instantaneous state — every deployment passes through Degraded and OutOfSync briefly. The Prometheus alerting chapter makes the general case for for: clauses.

Upgrades

   1. read the release notes. Argo CD's are unusually
      good and genuinely list breaking changes.
   2. back up: kubectl get applications,appprojects
      -n argocd -o yaml > backup.yaml
      (or, better: they are already in git)
   3. upgrade one minor version at a time
   4. upgrade the CLI to match — a version mismatch
      produces confusing API errors

Two specific things that have caught people:

Do not skip minor versions. CRD schema changes are applied per-version and skipping can leave resources that fail validation.

If Argo CD manages itself, upgrading is a commit — and if that commit breaks Argo CD, it cannot sync the fix. Chapter 14 covers the recovery, which is kubectl apply by hand. Knowing that in advance is the difference between five minutes and an hour.

Backup and restore

There is nearly nothing to back up, and that is the point:

argocd admin export > argocd-backup.yaml    # everything
argocd admin import - < argocd-backup.yaml

But the better answer is that Applications, AppProjects and configuration should be in git, at which point restoring Argo CD is: install it, apply one bootstrap Application, wait.

What is not in git and must be backed up separately:

• repository and cluster credentials (chapter 9) — unless generated by ESO from an external store
• the argocd-secret server key, if you use local accounts or long-lived API tokens
• any Sealed Secrets sealing key, which is not Argo CD’s but will end your day

Security hardening

A short list worth doing on day one:

Disable the local admin account once SSO works (admin.enabled: false).

Leave exec.enabled: false. It is a shell in production through Argo CD’s credentials (chapter 10).

Do not expose the UI publicly without SSO and, ideally, without a network path from the internet at all. It holds credentials to every cluster you manage.

Run the repo server with a restricted profile. It executes code from your repositories; it should have no service account token mounted and no network access it does not need.

Enable Kubernetes audit logging on Argo CD’s namespace. get secrets there is the event you want a record of.

Encrypt etcd at rest (the etcd chapter), because that is where all of these Secrets live.

What to take from this chapter

• Configure webhooks. It fixes latency (180 s → 1 s) and git-host rate limiting, then raise timeout.reconciliation so polling is a safety net.
• Exclude Events and EndpointSlices (and mesh/CNI per-pod resources) with resource.exclusions — the controller caches every resource in every managed cluster, and this is the biggest memory win.
• Shard the controller with controller.replicas and ARGOCD_CONTROLLER_REPLICAS — forgetting the second silently strands clusters. Use round-robin. Sharding only helps with multiple clusters.
• No CPU limit on the controller — reconciliation is bursty and throttling makes it fall behind.
• The repo server is stateless and easy to scale, but a per-repository lock means a monorepo does not benefit much from replicas.
• Turn on ApplyOutOfSyncOnly=true and ServerSideApply=true on large Applications.
• Notify on failure only. on-sync-failed and on-health-degraded.
• Watch argocd_app_reconcile p99 as the scaling signal; alert on duration, not instantaneous state.
• Upgrade one minor version at a time, and know that if Argo CD manages itself, a bad commit means recovering by hand.
• Almost nothing needs backing up if Applications and projects are in git — except credentials and any sealing key.
• Day-one hardening: disable local admin, leave exec off, do not expose the UI, restrict the repo server, audit get secrets, encrypt etcd.

Next: failure modes — stuck syncs, self-heal fights, the Argo CD that cannot fix itself, and the deletions that were not meant to happen.