We're rebuilding our cloud infrastructure and a Kubernetes cluster running a few thousand containers. The current network has separate VPCs for each resource type, with traffic between them going through a transit gateway.
We're using charts with Argo CD, splitting some observability components into separate namespaces, and considering Cilium with AWS IP allocation. What would you prioritize when setting this up again?
Start with how the platform will be operated, not just what the finished architecture looks like. Decide who owns each part, put configuration under GitOps, establish resource defaults, and test the cluster upgrade path before it becomes urgent.
Keep the initial design as simple as you can. Extra layers are easier to add later than to untangle once every workload depends on them.
Get observability in place early. Metrics, logs, and traces are easy to postpone until something breaks, which is exactly when you need them most.
Separate namespaces for those components can make sense, but also think about how you'll keep the signals useful without collecting a pile of duplicate or unnecessary metrics.
I'd start with default deny network policies and add explicit allows as services are introduced. Figuring out every dependency after hundreds of services are already talking is a much tougher cleanup job.
Same idea for RBAC. Scope permissions to namespaces where practical instead of leaving broad access that has to be tightened later.
Watch IP capacity if you go with Cilium in AWS ENI mode. Per-node ENI and prefix delegation limits can constrain pod density before CPU or memory does. Pods stuck in ContainerCreating can be a clue, so include IP allocation in capacity planning and troubleshooting.
Consider the cluster boundaries as well as namespace boundaries. One option is a cluster per environment, with a separate observability cluster collecting metrics and logs. That adds another cluster to operate, so weigh it against the isolation you actually need.
Answers are reviewed before publication.
_softly~misplaced_2:
Namespace boundaries deserve some thought too. Splitting by component can be useful, but a very fine-grained layout also means more RBAC and policy to maintain.