What Advantages Do Cilium and Calico Provide Over AWS CNI for EKS?
Asked by
TechLover101
I'm diving into different CNIs for Kubernetes and weighing the pros and cons for our two EKS clusters, each with around 5 nodes. I know AWS CNI integrates super smoothly with EKS, and it allows pods to be directly exposed on a private VPC range while utilizing security groups. However, one downside I've noticed is that IP exhaustion happens really quickly, even after we've tried prefix delegation and using larger instance types.
On the other hand, I've heard that Cilium and Calico could help with IP exhaustion and provide vendor-agnostic communication within the cluster. But what I've found is that they're not as tightly integrated with AWS, and I'm unsure if there's more to their disadvantages.
I'm currently using a Tailscale router in our cluster to connect to the Kubernetes API and I'm curious if I can still create a shell for a pod using Tailscale while using Cilium or Calico. Also, I'd love to hear from anyone with experience about what operational overhead I might be missing out on by not using AWS CNI for EKS. Thanks!
Answers
You might want to consider using AWS CNI's Custom Networking to help with the IP exhaustion issues you're facing. If you're interested in the features of Calico or Cilium, you can actually run both AWS CNI and Calico or Cilium together. It's a common setup in production environments, so you're not alone in this!
Reply to this answer
Wait, you’re running into IP exhaustion with just two clusters of 5 nodes? That seems a bit surprising. Are you using instance types like m5.xlarge? They have a cap on how many pods you can run per node without prefix delegation.
CloudWhisperer: Yeah, it can happen! I mean, when I was working with Azure Kubernetes Service, the CNI needed IPs from the VNet for every pod. With that setup, even a small cluster can hit limits pretty fast.
TechLover101: I should have clarified—I meant we're limited in pods per node without using larger instances or prefix delegation. It’s not exactly IP exhaustion on the subnet level.
Reply to this answer
Cilium and Calico both come with eBPF data planes, which can really boost performance, especially when scaling. Plus, they offer advanced security and observability tools that give you more control compared to standard AWS CNI. Just wanted to let you know that they can add a lot of features you won’t find in the native options.
Reply to this answer
Just a heads up, AWS EKS doesn’t officially endorse CNIs other than VPC CNI for standard operations, except for Hybrid nodes that may default to Cilium. If you're dealing with production workloads, you might get better support sticking with what AWS recommends. I did get Cilium running on EKS without major issues, but went back to Linkerd for better support and visibility.
Reply to this answer
Cilium offers this awesome feature called Hubble that lets you visualize all network flows within namespaces. This kind of insight isn’t something you typically get with vanilla EKS setups and it's super valuable for securing a large number of services.
Reply to this answer
Add your answer
Answers are reviewed before publication.
KubeNewbie42: I didn’t know CNIs could work together like that! I’ve only been working with Kubernetes for about half a year, so I appreciate the insight.
DevOpsDynamo: Exactly! We have over 20 EKS clusters running AWS CNI Custom Networking alongside Cilium, and it’s been a solid setup.