Switching to Cilium with Cluster-Pool Mode: Need Advice

Hey folks! We're looking into transitioning to Cilium as our CNI without kube-proxy, specifically using IPAM in "cluster-pool" mode due to limited IPv4 addresses in our company network. Essentially, we want nodes to have VPC-routable IPs while having Pods routed via the Cilium agent on an overlay network to help with IP consumption.

So far, it's been going fairly well, but we're running into a challenge. Since the EKS managed control plane doesn't detect the Pod Network, we need to expose any services that use webhook callbacks (like admission and mutation webhooks) through the hostNetwork of the node. This is especially important for deployments across the entire cluster such as the aws-lb-controller, kyverno, and cert-manager.

We figured once we sorted the port mappings on the nodes, we'd be set, but it's become pretty complicated. There are more services than we anticipated, and we had to adjust various container ports (metrics, probes, etc.), which isn't straightforward as many Helm charts don't let you change these parameters easily. We've had to do some post-rendering tweaks to make things work.

Now we've found out that tools like Crossplane come with their own webhooks for each provider, and we're starting to question if all the hassle with hostNetwork is worth it. I'm curious if anyone has navigated this route with Cilium and can share their experiences, especially if you've taken it to production!