Skip to content
Toolforge

10. Multi-cluster and tenancy

One Argo CD managing thirty clusters looks like an obvious win. It is also a single component whose compromise reaches thirty production environments. This chapter is about the machinery that makes multi-tenancy safe and the architectural question that machinery cannot answer.

AppProject — the actual security boundary

Section titled “AppProject — the actual security boundary”

Every Application belongs to an AppProject. There is a default project that permits everything, and leaving applications in it is the most common Argo CD security mistake.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop
  namespace: argocd
spec:
  description: The shop team


  # WHERE manifests may come from
  sourceRepos:
    - https://github.com/acme/shop-manifests.git
    - https://charts.bitnami.com/bitnami


  # WHERE they may be deployed
  destinations:
    - server: https://kubernetes.default.svc
      namespace: shop-*
    - name: prod-eu
      namespace: shop


  # WHICH cluster-scoped kinds are allowed (default: none)
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace


  # WHICH namespaced kinds are forbidden
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: rbac.authorization.k8s.io
      kind: ClusterRoleBinding


  # WHO may do what within this project
  roles:
    - name: developer
      policies:
        - p, proj:shop:developer, applications, get, shop/*, allow
        - p, proj:shop:developer, applications, sync, shop/*, allow
      groups:
        - acme:shop-developers

The four restrictions are the ones that matter:

sourceRepos — a team cannot deploy from a repository you have not approved. Without this, anyone who can create an Application can deploy anything from anywhere on the internet.

destinations — cluster and namespace pairs, with glob support. This is what stops the shop team deploying into kube-system.

clusterResourceWhitelist — empty by default, meaning no cluster-scoped resources at all. A project cannot create ClusterRoles, CRDs or Namespaces unless you allow it. This is the single most valuable default in Argo CD, and the reason a new project often “mysteriously” cannot create its own namespace.

namespaceResourceBlacklist — the specific escalations to forbid. ClusterRoleBinding is the important one: a team that can create one can grant itself cluster-admin, which makes every other restriction here decorative.

A project without clusterResourceWhitelist and a ClusterRoleBinding blacklist is not a boundary. Those two together are what makes an AppProject a security control rather than an organisational label.

Argo CD RBAC

Section titled “Argo CD RBAC”

Separate from Kubernetes RBAC and from projects. Configured in argocd-rbac-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
data:
  policy.default: role:readonly


  policy.csv: |
    # a role
    p, role:shop-admin, applications, *, shop/*, allow
    p, role:shop-admin, applications, delete, shop/*, deny
    p, role:shop-admin, logs, get, shop/*, allow
    p, role:shop-admin, exec, create, shop/*, deny


    # bind an SSO group to it
    g, acme:shop-leads, role:shop-admin


    # deployers may sync but not change definitions
    p, role:deployer, applications, sync, */*, allow
    p, role:deployer, applications, get, */*, allow

The format is Casbin: p, subject, resource, action, object, effect.

Resources: applications, applicationsets, clusters, projects, repositories, accounts, certificates, gpgkeys, logs, exec.

Three things to get right:

policy.default: role:readonly, never role:admin. An authenticated user who matches no rule should be able to look, not act.

exec is a separate permission and it is dangerous. argocd app exec opens a shell in a managed pod, and it is off by default (exec.enabled). Turning it on gives anyone with the permission a shell in production, through Argo CD’s credentials, bypassing your cluster’s own RBAC and audit. Leave it off, or deny it explicitly per role.

logs is also separate. Logs frequently contain secrets. Granting get logs on */* is a broader disclosure than it looks.

Deny always wins, regardless of order — so the deny on delete above holds even though the line above it allows *.

SSO

Section titled “SSO”

Two options:

# argocd-cm — direct OIDC, the simpler path
data:
  url: https://argocd.acme.com
  oidc.config: |
    name: Okta
    issuer: https://acme.okta.com
    clientID: xxx
    clientSecret: $oidc.okta.clientSecret     # from argocd-secret
    requestedScopes: ["openid", "profile", "email", "groups"]
    requestedIDTokenClaims: {"groups": {"essential": true}}

The groups claim is what the g, lines in the RBAC config bind to. If groups are not in the token, every user falls through to policy.default — which is the answer to “SSO works but nobody has any permissions”.

Use Dex only when your provider cannot speak plain OIDC to Argo CD directly — SAML, LDAP, or GitHub organisation membership.

Disable the local admin account once SSO works (admin.enabled: false). A shared local admin password is the credential that will still be in a wiki in three years.

Sync windows

Section titled “Sync windows”

Time ranges when syncing is allowed or denied:

spec:
  syncWindows:
    - kind: deny
      schedule: '0 0 * * *'        # cron: start
      duration: 24h
      applications: ['*-prod']
      manualSync: true             # humans may still sync
    - kind: allow
      schedule: '0 9 * * 1-5'      # weekdays 09:00
      duration: 8h
      applications: ['*-prod']

The semantics catch people: if any allow window exists for an application, syncing is denied outside all of them. An allow window is not additive permission on top of an open default; it closes everything else.

manualSync: true is the important escape — it lets a human sync during a deny window while auto-sync stays blocked. Without it you have built a change freeze you cannot break during an incident.

Sync windows are a genuinely good fit for change-freeze policies. They are a poor fit for “we only deploy on Tuesdays” as an engineering practice, but that is not Argo CD’s argument to have.

Apps in any namespace

Section titled “Apps in any namespace”

By default, Applications live only in argocd. That means every team’s Applications are in one namespace, and Kubernetes RBAC there is all-or-nothing — so project-based RBAC is the only separation.

Apps-in-any-namespace changes that:

# argocd-cmd-params-cm
data:
  application.namespaces: "team-*,shop"
# and the project must opt in
spec:
  sourceNamespaces:
    - team-*

Now a team creates Applications in its own namespace, with ordinary Kubernetes RBAC controlling who may do so. This is a genuine improvement for large installations — teams manage their own Applications with the RBAC they already have, and no Argo CD RBAC entry is needed per team.

Both settings are required. Allowing the namespace without adding it to the project’s sourceNamespaces produces an Application that is silently ignored, which is a confusing first experience.

The blast-radius question

Section titled “The blast-radius question”

The architectural decision this chapter builds up to, which the machinery above does not decide for you.

   ONE ARGO CD, MANY CLUSTERS


   ✓ one UI, one place to look
   ✓ one set of projects and RBAC
   ✓ easy cross-cluster views
   ✗ credentials to every cluster in ONE namespace
   ✗ its outage stops deployment everywhere
   ✗ its compromise reaches every environment
   ✗ the controller is a scaling bottleneck


   ONE ARGO CD PER CLUSTER


   ✓ NO cross-cluster credentials anywhere
   ✓ an outage affects one cluster
   ✓ scales trivially — each instance is small
   ✓ prod's Argo CD is not reachable from dev
   ✗ N UIs, N configurations
   ✗ no single view of everything
   ✗ N upgrades

The honest position: the per-cluster model is underrated and the centralised model is chosen too often for reasons of convenience rather than architecture.

The convenience is real — one UI genuinely is nicer. But look at what the centralised model concentrates. One namespace holds admin credentials to production in every region. Its compromise is not “an incident”; it is every environment at once. And the mitigation people reach for — AppProjects — is defence in depth inside a component that already holds the credentials, not a boundary around them.

A reasonable middle position, and the one most large organisations converge on:

   one Argo CD per ENVIRONMENT TIER
     - a dev/staging instance managing those clusters
     - a production instance managing production only,
       in a separate management cluster, with its own
       SSO groups and no path from the lower tiers


   → prod credentials are never in the same place as
     the environment where risky changes are tested

This keeps most of the convenience and removes the failure that actually hurts. Whatever you choose, choose it deliberately — the default of “add every cluster to the one we already have” is a decision being made by momentum.

What to take from this chapter

Section titled “What to take from this chapter”
  • AppProject is the real security boundary. Leaving Applications in default is the most common Argo CD security mistake.
  • Four restrictions matter: sourceRepos, destinations, clusterResourceWhitelist (empty by default — the best default in the tool), and namespaceResourceBlacklist with ClusterRoleBinding in it. Without the last two, a project is not a boundary.
  • Argo CD RBAC is separate from Kubernetes RBAC. Set policy.default: role:readonly; deny always wins.
  • exec is off by default — leave it off. It is a shell in production through Argo CD’s credentials, bypassing your cluster’s RBAC. logs is a real disclosure too.
  • SSO needs the groups claim; without it everyone falls through to the default role. Disable the local admin account afterwards.
  • Any allow sync window closes all other times. Set manualSync: true or you cannot deploy during an incident.
  • Apps-in-any-namespace requires both application.namespaces and the project’s sourceNamespaces, and lets teams use ordinary Kubernetes RBAC.
  • The blast-radius decision is architectural, not configurational. One Argo CD per cluster removes cross-cluster credentials entirely; one per environment tier is the pragmatic middle. Do not centralise by momentum.

Next: ApplicationSets — generating Applications instead of writing them, and the app-of-apps pattern for bootstrapping a cluster from nothing.

Written by Sepuri Sai Krishna

© 2026CC BY 4.0Source