Should I enable egress masquerading with Cilium in EKS when there’s a managed VPC gateway?

Pod networking often uses non-routable addresses, making it necessary for an overlay CNI, which helps manage large clusters. Masquerading allows pods to egress using the node's IP, which is where confusion can arise.

We opted to disable egress NAT/masquerading and let our VPC NAT Gateway manage egress traffic for our cluster. We're running IPAM in ENI mode on EKS without the kube-proxy. When we switched to this setup, we had to restart all services and pods that make outbound connections, like to the AWS API, as the clients weren't re-establishing TCP connections gracefully.

In ENI mode, each pod has its own interface for its assigned IP. You need to determine which interface should be used for outbound traffic that's routed outside the cluster, rather than between pods.

From what I've gathered, the VPC NAT gateway primarily deals with traffic going out to the internet, but it doesn't manage routing from pods to the VPC itself. The masquerading at the node level allows pod traffic to appear as though it’s coming from the node’s IP, which helps maintain proper return routing. Otherwise, you could experience some odd routing issues. That said, if you're using ENI mode and your pods already have routable VPC addresses, a lot of users actually choose to disable it since it may not be needed.