Hey everyone! I've built on-prem Kubernetes clusters using kubeadm, and now I'm diving into setting up a dev EKS cluster on AWS. Since I'm new to EKS, I have a couple of questions about networking that I hope you can help with. My plan is to construct everything manually via the AWS console before automating with Terraform.
First off, I need to plan my pod networking and IP allocations carefully to avoid IP exhaustion issues, especially because EKS assigns pod IPs directly from the VPC subnets. Here's my initial VPC plan:
- VPC CIDR: 10.16.0.0/16
- Availability Zones: 2 (eventually for all subnets)
- Public Subnets: 10.16.0.0/24 and 10.16.1.0/24 (for ALBs/NLBs and NAT gateways)
- Private Subnets: 10.16.16.0/20 and 10.16.32.0/20 (for worker nodes and pods)
Now, regarding a couple specific questions:
1. Will EKS assign pod IPs from the same subnet as the node's primary ENI?
2. I've experienced IP exhaustion when testing smaller subnets. How can I separate node and pod CIDRs in EKS?
3. Is the EKS CNI Custom Networking only useful in clusters that have IP limitations, or can I implement it from the start?
4. Should I associate extra subnets with the node group right away, using custom ENIConfigs to route pod IPs separately?
5. Regarding load balancing and ingress, I'd like to know about using dedicated external private IPs for certain services and how to handle that when multiple services need them.
Any advice would be appreciated as I'm keen to get this right from the get-go!
3 Answers
I’ve gone through similar setups before! First off, yes, EKS does assign pod IPs from the same subnet as the node's primary ENI. It can make things easier to manage if you start with larger CIDRs instead of worrying about managing additional subnets later. You can definitely use the EKS CNI Custom Networking from day one—you don’t have to wait until you're running into IP exhaustion. Also, check out Karpenter for scaling your nodes; it’s super useful, especially for dev environments. For your load balancing needs, using AWS's native ALB/NLB should cover what you mentioned about dedicated IPs without needing to maintain IP pools.
I totally get your concerns about the subnet setup. You can indeed create a VPC without public/private subnets initially and later deploy worker nodes to private subnets while using the VPC-CNI for pod networking. This way, you can keep managing your network resources easily. As for using External Secrets Operator, it’s a great choice for managing secrets with AWS Secrets Manager.
Have you looked into using RFC6598 subnets? I’m using 100.64.0.0/10 and it works great for allocating pod and node IPs. Deploying an ENIConfig resource can help you specify which subnets your pods should use. Plus, if you go for prefix assignment, the number of pods per node won't be limited by instance types anymore. For load balancing, I've found that Traefik combined with AWS Load Balancer Controller is a solid combo. It automates NLB creation and can simplify your setup.
Related Questions
How To Get Your Domain Unblocked From Facebook
How To Find A String In a Directory of Files Using Linux