I'm diving into the world of Kubernetes CNIs and I'm curious about their pros and cons, especially since we have two EKS clusters with around 5 nodes each running right now. With AWS CNI, I love the native integration with EKS, the ability to expose pods directly on our private VPC range, and the option to use security groups for pods. However, I'm running into some issues like IP exhaustion that happens way quicker than expected. We tried to fix this with prefix delegation and bigger instances, but we're still not actively monitoring our IP usage.
On the other hand, I've heard that Cilium or Calico could help with IP exhaustion and provide a vendor-agnostic way to communicate within the cluster. But I also know that they might not integrate as seamlessly with AWS. One of my concerns is whether I can still create a shell for a pod inside the cluster with Tailscale when using either Cilium or Calico since I'm using k9s. Am I missing anything critical? Any insights on the operational challenges of shifting away from AWS CNI for EKS?
6 Answers
With Cilium, one of the great features is Hubble, which shows all network flows in your namespaces. This tool is pretty invaluable when you have many services running and need to secure everything. It's something you won't find with a vanilla EKS setup.
You could use Cilium or Calico without having to break the bank. They're actually quite feature-rich compared to the AWS CNI.
Cilium and Calico's eBPF data planes can boost performance a lot, especially at scale. Plus, they provide robust security features and observability tools, which are definitely superior to the native options. That said, I’m not heavily into EKS, so just a heads up that your mileage may vary.
To tackle IP exhaustion, have you considered using AWS CNI Custom Networking? You can actually run AWS CNI alongside Cilium or Calico, which is a solid approach for production setups. That way, you can enjoy the benefits of both worlds.
Totally agree! We manage over 20 EKS clusters, and running AWS CNI Custom Networking with Cilium has been a game changer for us.
It's important to note that EKS doesn't officially support CNIs that replace the VPC CNI, other than the Hybrid/Anywhere node setups which might include Cilium. If you're having network issues, just be aware that AWS support might not help you out with alternatives outside of their recommended setups.
Wait, are you really hitting IP exhaustion with just two clusters of 5 nodes? That's surprising. From my experience in Azure Kubernetes, running a similar setup could definitely lead to faster exhaustion, but it seems a bit odd for EKS. How are you allocating your IPs?
Yeah, it can happen! Just to clarify, without prefix delegation and nitro instances, there are limits on how many pods you can have per node. I think you might be hitting those limits, not just subnet exhaustion.
Got it, thanks for explaining that! Makes sense that limit could sneak up on you.
I had no idea CNIs could work together! I’m still new to Kubernetes, so thanks for that info.