I've started a new role where our team is re-platforming the cloud infrastructure after inheriting some unusual design decisions, including separate VPCs for nearly every resource type and transit-gateway routing between them. We're effectively building the platform from scratch, including a Kubernetes cluster that will run several thousand containers.
I've worked with Kubernetes for about a year, but I'm looking for practical lessons from people who have operated large clusters and had a chance to rebuild them. What practices, policies, or architectural choices would you prioritize the second time around?
We're already using more namespaces and reducing the number of unrelated workloads grouped together. For example, observability is being split into separate namespaces for logs, metrics, and traces. We're also moving toward Cilium with AWS IP allocation, and the team is using Helm charts managed through Argo CD.
5 Answers
Using separate clusters per environment, with an additional observability cluster, can work well when the isolation and operational overhead are justified. I’d avoid splitting namespaces too aggressively at the start, though. Separate namespaces for logs, metrics, and traces are reasonable, but creating one for every team-and-environment combination can multiply RBAC, networking, and deployment maintenance. Start with clear coarse boundaries and split them when there’s a concrete need.
Optimize for operability rather than designing the most elaborate architecture. Establish GitOps, clear ownership boundaries, sensible default resource requests and limits, and a well-tested cluster upgrade process early. It’s usually easier to add complexity later than to remove it from a running platform.
Start with default-deny NetworkPolicies and tightly scoped, namespace-level RBAC. Adding explicit permissions as services are introduced is much easier than trying to understand and restrict hundreds of existing service relationships later. The same principle applies to ownership and access boundaries.
Build observability near the beginning instead of waiting for the first major incident. Make sure the team can quickly answer what is failing, which workloads are affected, and whether the problem is application, node, networking, or control-plane related. Keep an eye on noisy, duplicated, or unnecessary metrics so the system stays usable as it grows.
Be careful with Cilium on AWS, especially when using ENI or prefix-based IP allocation. A cluster running thousands of containers can hit per-node or ENI address limits before CPU or memory becomes a problem. The symptom may simply be pods stuck in ContainerCreating, so monitor IP capacity and test scaling behavior before production.
We’re using Helm charts with Argo CD already. I joined a small team that has some plans in place, so I’m mainly trying to contribute without adding unnecessary complexity.