Hi all, I'm facing a challenge with our Kubernetes cluster regarding egress traffic for a group of pods. We need to ensure two things: first, that the original source port is preserved when these pods send outbound traffic (meaning we want to avoid any SNAT port rewriting). Second, we want to maintain a consistent source IP address across all nodes, so every pod uses the same egress IP, regardless of their scheduling location. We've explored the Cilium Egress Gateway but found it tricky to achieve a reliable egress IP across multiple nodes. Additionally, Cilium's eBPF-based masquerading tends to alter the source port, which is something we need to avoid. If anyone has dealt with a similar situation or knows how to achieve both a static egress IP across nodes and preserve the original source port, your tips or examples would be greatly appreciated! Thanks!
3 Answers
Have you considered using a proxy setup? If your web service requires it, building a proxy can help manage what you need before connecting to the external system. That way, you can control the IP and port details more effectively.
You might want to look into using an Istio Egress Gateway. It can give you the tools you need for managing egress traffic while retaining the source IP and port. Check out the documentation for a deeper dive: https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/
Cilium does offer a similar egress gateway feature that's integrated into its CNI, which might be easier to set up since it's all included without extra deployments. Just a note though, the free version doesn’t offer high availability.
It sounds like you might want to explore netgraph as a possible solution for your network management needs within Kubernetes. Have you considered the overall architecture of your Kubernetes setup?
We're in a bit of a bind with this because we rely on a legacy external system that demands the original source IP and port. Unfortunately, we're unable to change their validation checks.