We currently operate around 100 customer-site Kubernetes clusters and want an architecture that can grow beyond 1,000 without becoming unmanageable. The fleet is mostly small Linux-container clusters, with a mix of single-node and highly available deployments. Connectivity varies from fully connected to heavily restricted or only occasionally online.
We already use Argo CD and prefer open-source GitOps and infrastructure-as-code tools. We are evaluating options such as Rancher/RKE2, Talos, Cluster API, Kubermatic, Fleet, and Sveltos.
For people running similar fleets in production, what has worked for cluster provisioning, lifecycle management, upgrades, disconnected sites, GitOps architecture, and customer-specific configuration and secret isolation? Do you use one central management system, an agent or GitOps controller per cluster, or some hybrid? What problems only become obvious once you reach hundreds or thousands of clusters? The application is largely the same everywhere, although versions and feature settings may differ. I'm especially interested in real operational experience rather than vendor recommendations.
5 Answers
A common pattern is Cluster API for provisioning and lifecycle management, with clusters grouped by environment or other meaningful similarities. GitOps can then apply the appropriate configuration, while SOPS or an equivalent encryption workflow keeps customer secrets isolated. At that point, operating 400 clusters is conceptually similar to operating 40, provided the reconciliation and update processes are fully automated.
The difficult part is not usually creating the first thousand clusters; it is safely changing them later. Kubernetes upgrades, CNI or CSI replacements, security policy changes, and application rollbacks need staged rollouts, health checks, pause controls, and a way to handle clusters that are offline for weeks. Homogeneous clusters and feature flags help a lot, but you still need clear version support and recovery procedures.
Keep the customer boundary explicit. Namespace isolation inside a shared cluster may be cheaper, but it is not equivalent to a dedicated cluster when customers require infrastructure-level separation or operate in their own facilities. For dedicated clusters, use separate credentials, encrypted per-customer configuration, network and firewall boundaries, and independent recovery paths. Avoid forcing every site through one always-online management plane; eventually it is usually simpler to let each cluster manage itself and use the central system mainly for desired state, inventory, and rollout coordination.
Gardener is worth evaluating for this scale. It is designed around managing large numbers of workload clusters and can provide a consistent lifecycle model across infrastructure providers. The initial setup and production hardening are more involved than a basic Rancher installation, but that investment can pay off once the fleet becomes very large. Talos combined with Cluster API is another strong option when immutable nodes and consistent upgrades are priorities.
The trade-off with Gardener is the learning curve and the complexity of building the first production-ready management environment. It becomes more attractive when you genuinely need hundreds or thousands of clusters rather than only a few dozen.
For large or unreliable fleets, pull-based management tends to be more resilient than a central controller maintaining a live connection to every cluster. Flux-style agents or a separate Argo CD instance per cluster create stronger failure boundaries and work better when sites are intermittently connected. A central hub can handle registration and policy, while each cluster performs its own reconciliation locally.
Central Rancher and Argo CD installations have reportedly become difficult to operate beyond roughly 500 connected clusters in some deployments. Agent-based approaches may use more resources overall, but they avoid turning one management cluster into a bottleneck or single point of failure.
The main design question is how updates are fanned out when every cluster has its own repository. A shared repository with cluster- or environment-specific overlays can reduce that duplication, but it still needs careful rollout controls.