We currently operate around 100 customer-site Kubernetes clusters and want an architecture that can grow beyond 1,000 without becoming unmanageable. The clusters are generally small, with a mix of single-node and highly available setups. Some sites have reliable connectivity, while others are heavily restricted or only connect intermittently. They run Linux containers and deploy essentially the same application, although versions and configuration may differ by customer.
We already use Argo CD and prefer open-source GitOps and infrastructure-as-code tools. We are evaluating options such as Rancher/RKE2, Talos, Cluster API, Kubermatic, Fleet, and Sveltos.
I'm interested in real production experience with cluster provisioning, lifecycle management, upgrades, centralized versus per-cluster GitOps controllers, disconnected or air-gapped sites, customer-specific configuration and secret isolation, and the operational problems that appear at several hundred or several thousand clusters. Practical lessons about what failed would be especially useful; this isn't intended to be a vendor-pitch discussion.
5 Answers
For a homogeneous fleet, the biggest advantage comes from standardization and automation rather than a particular product. Use immutable or tightly controlled node images, fully automated cluster creation, explicit version channels, and continuous drift correction. Feature flags and per-customer values let the same application layout serve many sites without creating a unique deployment model for every customer.
Be careful with Git itself at this scale. Thousands of clusters polling large repositories can put substantial load on the Git service. Stagger reconciliation, tune intervals, keep repositories small, or publish versioned manifests as OCI artifacts or objects in internal storage when appropriate.
The recurring pain is maintenance, not initial provisioning. A fleet can be created with Terraform, Ansible, Cluster API, or similar tools, but upgrades become difficult when a new Kubernetes release or system component must roll out across hundreds of sites with different connectivity, hardware, and local constraints. Build progressive rollout waves, health checks, automatic pause conditions, and a tested rollback or recovery path before the fleet becomes large.
Also avoid assuming a single central platform team can absorb every exception. Standardize the supported cluster profiles and make deviations explicit. Once every customer has a special networking, storage, or upgrade requirement, the number of clusters matters less than the number of unique operating models.
A common pattern is a management or hub cluster that runs the provisioning and policy tools, while the customer clusters reconcile their own desired state. Cluster API works well for lifecycle management, and GitOps can handle the workloads and configuration. Some teams keep one repository with cluster and environment conditions; others use separate repositories for stronger boundaries.
For isolation and resilience, running an agent or GitOps controller in each downstream cluster is often more reliable than having one central controller maintain hundreds of persistent connections. Separate Argo CD instances per cluster provide strong failure boundaries, while pull-based tools such as Flux can scale more naturally because each cluster works independently. The main operational difficulty is coordinating disruptive changes, such as Kubernetes, CNI, CSI, or application upgrades, and safely rolling them back across clusters running different versions.
For restricted or offline locations, make the cluster operationally independent. It should have the images, charts, manifests, certificates, and upgrade artifacts needed for normal operation in a local registry or cache. A pull-based agent is generally a better fit than requiring the management plane to reach every site continuously. Treat connectivity as an optimization, not as a requirement for the cluster to function.
Customer isolation should be enforced at more than just the application layer: separate credentials and repositories or paths, narrowly scoped service accounts, per-customer secrets encrypted with tools such as SOPS, and network and policy boundaries. If the customer requirement is genuine infrastructure isolation, namespaces in a shared cluster usually won’t provide the same assurance as separate clusters.
Gardener is worth evaluating for this size of fleet. It is designed around managing large numbers of workload clusters and can fit well with multiple infrastructure providers and GitOps workflows. The initial architecture and production setup are more involved than a simple Rancher installation, but that investment can pay off when clusters are numerous and fairly standardized.
Other viable combinations include Talos with Cluster API, or Rancher/RKE2 for teams that want a more integrated management experience. The important question is where reconciliation happens and how much state the central management plane must maintain, especially for sites that are intermittently connected.
Central management tools can hit a practical limit well before the theoretical cluster count. In one large deployment, both the central dashboard and GitOps controller became difficult to operate beyond roughly 500 connected clusters. Moving more reconciliation into the clusters themselves reduced that dependency on the hub.