11. ApplicationSets and app-of-apps

Fifty microservices across four environments is two hundred Application resources. Writing them by hand is a copy-paste exercise that guarantees drift between them. Two patterns solve this, and they solve slightly different problems.

App-of-apps

The older, simpler pattern: an Application whose manifests are other Applications.

   Application "bootstrap"
     source: github.com/acme/manifests, path: bootstrap/prod
       │
       ├── applications/cert-manager.yaml   → Application
       ├── applications/ingress-nginx.yaml  → Application
       ├── applications/prometheus.yaml     → Application
       └── applications/shop.yaml           → Application

You create one Application by hand. It syncs a directory of Application manifests, Argo CD creates them, and each syncs its own workloads.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/manifests.git
    targetRevision: main
    path: bootstrap/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd            # Applications live here
  syncPolicy:
    automated: { prune: true, selfHeal: true }

This is the standard cluster bootstrap: a new cluster gets Argo CD and this one Application, and everything else installs itself. Ordering comes from sync waves on the child Applications:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # cert-manager before things needing certs

Its limitation is that every child is a file you wrote. Adding an application means adding a file; adding an environment means copying twenty files.

ApplicationSets

A controller that generates Applications from a template and a source of parameters.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: shop-services
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - name: checkout
            namespace: shop
          - name: catalogue
            namespace: shop
          - name: payments
            namespace: payments
  template:
    metadata:
      name: '{{name}}'
    spec:
      project: shop
      source:
        repoURL: https://github.com/acme/manifests.git
        targetRevision: main
        path: 'apps/{{name}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Three Applications from one resource. Add a fourth element, get a fourth Application.

The ApplicationSet controller ships with Argo CD (it was separate before 2.3) and owns the Applications it generates — so a generated Application edited by hand is reverted, exactly as a drifted Deployment would be.

The generators

list — a literal list. Explicit and readable; a good default when the set changes rarely.

cluster — one Application per registered cluster:

generators:
  - clusters:
      selector:
        matchLabels:
          env: prod

This is the reason many people adopt ApplicationSets. Register a new cluster with the right label and every platform component deploys to it automatically. Cluster labels and annotations are available as {{metadata.labels.region}} and similar.

git — directories. One Application per directory in a repository:

generators:
  - git:
      repoURL: https://github.com/acme/manifests.git
      revision: main
      directories:
        - path: apps/*
        - path: apps/experimental/*
          exclude: true

Adding a directory to git creates an Application. This is the most powerful and most dangerous generator in the list, because a merged pull request that adds a directory deploys to production with no further approval. That is either exactly what you want or a serious governance gap, and which one depends entirely on how your repository is protected.

git — files. One Application per matching config file, with the file’s contents as parameters:

generators:
  - git:
      repoURL: https://github.com/acme/manifests.git
      revision: main
      files:
        - path: 'apps/*/config.json'

{ "name": "checkout", "namespace": "shop", "replicas": 5 }

This is usually the best of the git generators: adding an application is adding a small declarative file that a reviewer can read, rather than a directory whose contents decide what happens.

matrix — the cross product of two generators. This is where the real power is:

generators:
  - matrix:
      generators:
        - clusters:
            selector: { matchLabels: { env: prod } }
        - git:
            repoURL: https://github.com/acme/manifests.git
            revision: main
            directories:
              - path: platform/*

Every platform component on every production cluster. Four components and five clusters is twenty Applications from one resource, and adding a cluster adds four.

merge — combines generators by a key, letting later ones override earlier values. The pattern for per-cluster overrides:

generators:
  - merge:
      mergeKeys: [server]
      generators:
        - clusters: {}                              # defaults for all
        - list:
            elements:
              - server: https://prod-eu.example.com
                replicas: "10"                      # override for one

pullRequest — an Application per open pull request, for preview environments. Genuinely useful and genuinely a footgun: every open PR becomes a running environment, and without requeueAfterSeconds tuning and a label filter you will discover how many stale PRs your organisation has.

scmProvider — one Application per repository in a GitHub or GitLab organisation, optionally filtered by whether a given file exists. For platform teams onboarding services automatically.

Templating

Beyond {{param}} substitution, ApplicationSets support Go templating, which is worth enabling:

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  template:
    metadata:
      name: '{{.name}}-{{.cluster}}'
    spec:
      source:
        helm:
          values: |
            replicaCount: {{ default 2 .replicas }}
            {{- if eq .env "prod" }}
            resources:
              requests: { cpu: "1", memory: "2Gi" }
            {{- end }}

Set missingkey=error. Without it, a typo in a parameter name renders as an empty string, and an empty string in the wrong field — a namespace, an image tag — deploys something unintended. This one option turns a silent misconfiguration into a loud failure.

The deletion behaviour everyone learns once

Delete an ApplicationSet and every Application it generated is deleted — and with the resources finalizer, so are all their workloads.

kubectl delete applicationset shop-services
# → 3 Applications deleted → every workload deleted

This is correct and consistent, and it has taken down more than one production environment.

The protections:

spec:
  syncPolicy:
    preserveResourcesOnDeletion: true    # keep resources if the AppSet goes
    applicationsSync: create-only        # or: create-update

applicationsSync is the important one and it is worth understanding as a spectrum:

Value	The ApplicationSet may
`create-only`	create Applications; never update or delete them
`create-update`	create and update; never delete
(unset)	create, update and delete

create-update is a good production default. Generated Applications stay in sync with the template, but removing an element from a generator does not delete a live application — it leaves it for a human to remove deliberately.

Also worth knowing: an ApplicationSet with a git-directory generator on a branch that is deleted generates zero Applications, which with default settings deletes everything. A CI job that prunes old branches can take down an environment. Guard against it with create-update.

Progressive syncs

For rolling a change across clusters in stages rather than everywhere at once:

spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: env
              operator: In
              values: [dev]
        - matchExpressions:
            - key: env
              operator: In
              values: [staging]
        - matchExpressions:
            - key: env
              operator: In
              values: [prod]

Each step waits for the previous group’s Applications to be Healthy. This is one of the more valuable ApplicationSet features and it is still marked experimental — it requires enabling and it does change behaviour, so test it somewhere unimportant first.

Choosing between them

   APP-OF-APPS when:
     ✓ the set of applications is small and stable
     ✓ each one is genuinely different
     ✓ you want every application visible as a file
     ✓ bootstrapping a cluster

   APPLICATIONSET when:
     ✓ many applications share a shape
     ✓ the set changes (new clusters, new services, PRs)
     ✓ you are deploying the same thing to many clusters
     ✓ you want onboarding to be adding one small file

They compose: an app-of-apps whose children include ApplicationSets is a common and good structure. The bootstrap Application creates the platform ApplicationSet, which generates per-cluster platform components, while genuinely bespoke applications stay as individual files.

The bootstrap ordering problem

A note for the cluster-creation case, because the ordering is subtle:

   1. cluster exists
   2. install Argo CD (helm, or kubectl apply — this
      one step is NOT GitOps, and cannot be)
   3. apply ONE bootstrap Application, pointing at
      the repository
   4. that Application includes an Application managing
      Argo CD ITSELF
   5. from now on, Argo CD upgrades come from git

Step 2 is the irreducible bit. Something outside GitOps has to create the first thing — usually Terraform that creates the cluster and installs the Argo CD Helm chart, or a script. The chicken-and-egg is real and there is no elegant escape; the goal is to make step 2 as small as possible so that everything after it is declarative.

Step 4 is the part worth doing and the part that goes wrong. Argo CD managing itself is genuinely useful, and chapter 14 covers the specific way it bites: a bad commit to Argo CD’s own manifests can leave you with an Argo CD that cannot sync the fix.

What to take from this chapter

App-of-apps is an Application whose manifests are Applications. It is the standard cluster bootstrap and it is fine when the set is small and stable.
ApplicationSets generate Applications from a template plus a generator, and own what they generate.
Generators: list (explicit), clusters (one per cluster — the usual reason to adopt them), git directories (adding a directory deploys to production), git files (usually the best git option), matrix (cross product — the real power), merge (per-cluster overrides), pullRequest, scmProvider.
Enable goTemplate with missingkey=error — otherwise a typo renders as an empty string into a namespace or image tag.
Deleting an ApplicationSet deletes every Application it generated and their workloads. Use applicationsSync: create-update in production so removal is never automatic, and know that a deleted git branch can generate zero Applications.
RollingSync stages a change across clusters, waiting for health at each step. Still experimental.
They compose: an app-of-apps whose children include ApplicationSets is a good structure.
The bootstrap chicken-and-egg is irreducible. Something outside GitOps installs Argo CD; make that step as small as possible, then have Argo CD manage itself.

Next: secrets — the problem Argo CD deliberately does not solve, and the four ways people solve it.

Written by Sepuri Sai Krishna