Hey folks! We're considering making the switch to Cilium as our CNI without kube-proxy, specifically using it in IPAM mode "cluster-pool" (instead of ENI). This change is prompted by limitations we have with usable IPv4 addresses on our company network. With this setup, only the nodes get VPC routable IPs while Pods are routed through the Cilium agent on the overlay network, which should help us reduce IP consumption significantly.
So far, it's been working okay, but we didn't anticipate a major drawback: since the EKS managed control-plane doesn't recognize the Pod network, we need to expose any services that use webhook callbacks (like admission and mutation) through the host network of the nodes. This mainly affects cluster-wide deployments such as aws-lb-controller, kyverno, and cert-manager. We thought that mapping these services with non-conflicting ports would solve our issues, but it turned out to be quite a headache. We also realized many Helm charts don't allow us to change all the necessary ports easily, forcing us to employ post-rendering workarounds.
Now we've discovered that some tools like Crossplane introduce their own webhooks for each provider, and we're starting to doubt if all this host network configuration is worth the trouble. I'd love to hear if anyone else has navigated this path with Cilium and what their experience has been, especially if you've taken this setup to production!
17 Answers
,
,
a
,
T
D
Absolutely! Instead of cluster-pool, I opted for ENI with prefix delegation. With that, your services don’t have to worry about the pod-network since it’s part of the VPC. I had to migrate away from using hostNetwork for the same reason; it just became a mess, especially since some applications didn’t support hostNetwork in their official Helm charts. I also disable hostport in Cilium and use IPv4 masquerade now. It’s much cleaner!
,
,
,
,
,
,
{
c
r
That makes sense! I see how prefix delegation can help with pod density, but it looks like it might fragment the IP space even more. That's why we aimed for the cluster-pool to tackle both pod density and IP address concerns. Could you share why you went for IPv4 masquerading in ENI mode? I’d thought the pods would be directly routable there.