Hey everyone! I'm working on organizing our AWS EKS setup, which currently includes two clusters: dev and production. My director has asked me to create two additional clusters for staging (QA) and corporate (internal use). I've got the Terraform code ready to go, but I need advice on the networking aspect. We're considering a separate VPC CIDR for each environment. In my last job, QA and production shared the same VPC CIDR for testing purposes—1% of traffic was routed to QA while using production's infrastructure. Is this a good practice, and what are the best approaches for setting up the network for these clusters?
5 Answers
Having one VPC CIDR per environment is definitely a good practice. You should avoid having production share a CIDR with any non-production environments; they should be isolated. Production traffic should never touch non-production setups, just to keep things secure and efficient.
Totally agree! It's crucial to keep production and testing environments separate to maintain integrity.
You should definitely aim for separate VPCs for each cluster. This minimizes the risk of running out of IPs as you scale. If your databases are outside the cluster, consider putting them in their own VPC and peering them with the Kubernetes VPC. For testing interactions between QA and PROD, looking into strategies like blue/green deployments or canary releases might be more beneficial than mixing environments.
The choice really depends on how many environments you plan to have and the number of pods you're anticipating. It will influence the network plugin you should use, which is essential for your networking setup to function effectively.
In AWS, a subnet doesn't control reachability; it's really just for IP allocation. If you want proper isolation between your clusters, you need separate VPCs. Otherwise, you'll risk unwanted communication between environments, like dev and production, which can be a disaster.
While it's not exactly your question, I would separate AWS accounts for production and non-production. Having different VPCs for staging and lower environments is a good practice. AWS generally recommends separate accounts per environment, but this approach balances costs and management.
Separate accounts definitely help with budget management as you scale up, plus it can keep things organized. Also, have you looked into EKS Auto mode? It simplifies management, though it does come with a compute up-charge.
Good point! Being cautious with IP space and network traffic costs is definitely a must as your setup grows.
Using API gateways or WAF for traffic routing instead of relying solely on the network is a solid strategy.