I'm trying to navigate my way through Kubernetes and need help managing a cluster with multiple gateways. I have several Kubernetes hosts, each connected to two networks with separate gateways. My goal is to configure specific pods in a certain namespace so that all their external traffic goes through one particular gateway.
The second gateway is set up in pfSense to route traffic through a VPN. I attempted to use a secondary interface (thanks to Multus) and set default routes for the external traffic, but it seems like DNS queries are still being handled internally, which isn't what I want. I also tried to direct all DNS queries through pfSense, but then the internal cluster DNS stopped working.
I feel like I might not be approaching this the right way. Any suggestions on how to get this architecture right?
1 Answer
Managing multiple gateways in Kubernetes can indeed be tricky. The solution might depend on the CNI (Container Network Interface) you're using. If it's Cilium, you could potentially set up an egress gateway. This way, only a specific node would have access to your secondary network, and you can define policies to target specific pods and namespaces. Check out the egress gateway documentation for detailed steps!
Thanks for the tip! I’m using Calico, so I’ll look into how egress gateways work without Cilium. I’ve got pods pinned to the host with the secondary interface, so hopefully that helps.