12. Secrets and image updates

Two problems that GitOps creates rather than solves. Both come from the same premise — that git contains the complete desired state — and both need an answer before a real deployment works.

The secrets problem

Argo CD applies what is in git. A Secret in git is a secret in git, base64-encoded, forever, in every clone, in every fork, in the history even after you delete it.

There is no configuration that fixes this. Argo CD does not manage secrets and does not intend to. You need one of four patterns.

1. Sealed Secrets

Encrypt with a public key; only the in-cluster controller holds the private key.

kubeseal --format yaml < secret.yaml > sealed-secret.yaml
git add sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: shop
spec:
  encryptedData:
    password: AgBv3x9K...        # only this cluster can decrypt

Good	Bad
genuinely GitOps — the secret is in git, safely	the controller’s private key is your single point of failure
no external dependency at runtime	encrypted per namespace+name; renaming means re-encrypting
simple to operate	no rotation — changing a value is a commit
free	lose the key, lose every secret

Back up the sealing key. It is one Secret in kube-system, and without it every SealedSecret in git is permanently undecryptable. This is the mistake that ends Sealed Secrets adoptions.

Best for: small teams, one or few clusters, secrets that rarely change.

2. SOPS with age or KMS

Encrypt values in-place; decrypt at render time, usually via a config management plugin or helm-secrets.

# values.enc.yaml — readable structure, encrypted values
database:
  host: postgres.shop.svc          # plaintext
  password: ENC[AES256_GCM,data:x8Kk...,type:str]

Good	Bad
diffs are readable — you see which key changed	needs a CMP or plugin in the repo server
keys in cloud KMS, so access is IAM-controlled and audited	the repo server needs decryption rights
works outside Kubernetes too	more moving parts
granular — encrypt only the values

That first row is the underrated advantage. A SealedSecret’s diff is one opaque blob changing; a SOPS diff shows that password changed and host did not.

Best for: teams already using cloud KMS, and anyone who cares about reviewable diffs.

3. External Secrets Operator

The one to choose if you are not sure. Git contains a reference; the operator fetches the value at runtime.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/shop/database
        property: password

Nothing secret is in git — not even encrypted. Git says “the value lives at this path”; the operator authenticates to the store and creates the Secret.

Good	Bad
no secret material in git at all	needs a secret store to run
rotation works — change it in the store, the operator picks it up	one more controller
central audit of who read what	a store outage means new pods cannot start
works with Vault, AWS/GCP/Azure secret managers, 1Password, and more

The rotation row is what separates this from the first two. With Sealed Secrets or SOPS, rotating a credential is a commit, a review and a sync. With ESO it is a write to the store, and the credential changes everywhere within refreshInterval.

Pair it with Vault’s dynamic secrets and the credential in the cluster was created for that consumer and expires on its own — which is a materially different security position from “a shared password, encrypted”.

Best for: most people. Especially anyone who already runs Vault or a cloud secret manager.

4. The Vault Agent injector / CSI driver

Skip Kubernetes Secrets entirely. The secret is injected into the pod’s filesystem by a sidecar or a CSI volume.

Good	Bad
never becomes a Kubernetes Secret at all	changes your pod spec
lease-based, renewed and revoked properly	the application must read from a file
finest-grained access control available	tightest runtime coupling to Vault

Best for: high-security environments where a Secret existing in etcd is itself unacceptable.

Comparison

	Sealed Secrets	SOPS	ESO	Vault injector
Secret material in git	encrypted	encrypted	none	none
Rotation without a commit	✗	✗	✓	✓
Readable diffs	✗	✓	✓	n/a
External dependency	none	KMS	store	Vault
Operational cost	lowest	medium	medium	highest
Audit of secret access	✗	KMS-level	✓	✓

Recommendation: External Secrets Operator, backed by whatever store you already have. It is the only option in the table where rotation is not a git commit, and rotation is the property that actually reduces the damage of a leak.

If you have nothing and want the smallest possible step: Sealed Secrets, with the sealing key backed up somewhere you will still have in two years.

The image update problem

The other awkwardness. Your CI builds acme/checkout:v1.4.2. Something must change git so Argo CD deploys it.

Option 1: CI writes the commit

# in the pipeline, after pushing the image
yq -i '.image.tag = "v1.4.2"' envs/prod/values.yaml
git commit -am "checkout v1.4.2" && git push

Good	Bad
explicit, obvious, in git history	CI needs write access to the manifest repo
works with any tagging scheme	commit noise
easy to review and revert	a write credential that can change production

That second row is worth pausing on, because it partially undoes the argument from chapter 1. You removed cluster credentials from CI and gave it a credential that can change what runs in production instead. It is still better — a commit is reviewable and revertible where a kubectl apply is neither — but it is not “CI has no production access”.

Mitigate it by keeping manifests in a separate repository with branch protection, so CI’s write access does not also mean write access to application source.

Option 2: Argo CD Image Updater

A separate controller watches registries and updates the tag itself.

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=acme/checkout
    argocd-image-updater.argoproj.io/app.update-strategy: semver
    argocd-image-updater.argoproj.io/app.allow-tags: 'regexp:^v[0-9]+\.[0-9]+\.[0-9]+$'
    argocd-image-updater.argoproj.io/write-back-method: git

Two write-back methods, and the difference matters:

git — commits the new tag to your repository. Git stays the source of truth. Requires Argo CD to have write access to your manifest repository, which is the thing chapter 9 said it never needs. This is the exception, and it is a deliberate trade.

argocd — changes the Application’s parameter override directly, without committing anything. Convenient, and it breaks the core promise: the cluster no longer matches git, and a git revert will not roll back. Do not use this in production.

Image Updater is genuinely useful for non-production environments — auto-deploying the latest build to dev with no pipeline work. In production it takes a deployment decision away from a reviewed commit, which is usually the wrong direction.

Option 3: a pull request

CI opens a PR rather than committing. A human merges.

Slowest and safest. For production changes this is what most regulated environments settle on, and the friction is the feature: every deployment has an approver and a record.

The recommendation

   dev/staging  → Image Updater, or CI commits directly
   production   → CI opens a PR, a human merges

And regardless of method: pin to immutable tags or digests. acme/checkout:latest in a GitOps repository is a contradiction — git no longer describes what runs, because the tag can move under you. A digest (acme/checkout@sha256:...) is the only fully honest option, and the Docker guide covers why a tag is a mutable pointer and a digest is not.

What to take from this chapter

• Argo CD does not manage secrets, deliberately. A Secret in git is in every clone and in history forever.
• Sealed Secrets: simplest, no external dependency, no rotation, and the sealing key is a single point of failure you must back up.
• SOPS: encrypted values with readable diffs and KMS-controlled access; needs a plugin in the repo server.
• External Secrets Operator: nothing secret in git at all, and rotation without a commit. The default recommendation, especially with Vault or a cloud secret manager behind it.
• Vault injector/CSI: the secret never becomes a Kubernetes Secret; highest security, tightest coupling.
• Rotation is the property that actually limits the damage of a leak, and only the last two have it.
• Image updates: CI commits (simple, but CI gets write access to manifests — keep them in a separate protected repository), Image Updater (good for dev; write-back-method: argocd breaks GitOps and does not belong in production), or a pull request (slowest, safest, right for production).
• Never :latest. Pin a version tag, and prefer a digest — a tag is a mutable pointer, and a mutable pointer means git does not describe what runs.

Next: tuning and operations — sharding the controller, the settings that matter at scale, and what to monitor.