I'm diving into different CNIs for Kubernetes and weighing the pros and cons for our two EKS clusters, each with around 5 nodes. I know AWS CNI integrates super smoothly with EKS, and it allows pods to be directly exposed on a private VPC range while utilizing security groups. However, one downside I've noticed is that IP exhaustion happens really quickly, even after we've tried prefix delegation and using larger instance types.
On the other hand, I've heard that Cilium and Calico could help with IP exhaustion and provide vendor-agnostic communication within the cluster. But what I've found is that they're not as tightly integrated with AWS, and I'm unsure if there's more to their disadvantages.
I'm currently using a Tailscale router in our cluster to connect to the Kubernetes API and I'm curious if I can still create a shell for a pod using Tailscale while using Cilium or Calico. Also, I'd love to hear from anyone with experience about what operational overhead I might be missing out on by not using AWS CNI for EKS. Thanks!
5 Answers
Cilium offers this awesome feature called Hubble that lets you visualize all network flows within namespaces. This kind of insight isn’t something you typically get with vanilla EKS setups and it's super valuable for securing a large number of services.
Just a heads up, AWS EKS doesn’t officially endorse CNIs other than VPC CNI for standard operations, except for Hybrid nodes that may default to Cilium. If you're dealing with production workloads, you might get better support sticking with what AWS recommends. I did get Cilium running on EKS without major issues, but went back to Linkerd for better support and visibility.
Cilium and Calico both come with eBPF data planes, which can really boost performance, especially when scaling. Plus, they offer advanced security and observability tools that give you more control compared to standard AWS CNI. Just wanted to let you know that they can add a lot of features you won’t find in the native options.
Wait, you’re running into IP exhaustion with just two clusters of 5 nodes? That seems a bit surprising. Are you using instance types like m5.xlarge? They have a cap on how many pods you can run per node without prefix delegation.
I should have clarified—I meant we're limited in pods per node without using larger instances or prefix delegation. It’s not exactly IP exhaustion on the subnet level.
You might want to consider using AWS CNI's Custom Networking to help with the IP exhaustion issues you're facing. If you're interested in the features of Calico or Cilium, you can actually run both AWS CNI and Calico or Cilium together. It's a common setup in production environments, so you're not alone in this!
I didn’t know CNIs could work together like that! I’ve only been working with Kubernetes for about half a year, so I appreciate the insight.
Exactly! We have over 20 EKS clusters running AWS CNI Custom Networking alongside Cilium, and it’s been a solid setup.
Yeah, it can happen! I mean, when I was working with Azure Kubernetes Service, the CNI needed IPs from the VNet for every pod. With that setup, even a small cluster can hit limits pretty fast.