Do I Need Egress Masquerading with a VPC NAT Gateway in EKS and Cilium?

Pods are usually non-routable and separate from the node network; that’s why we have overlay CNI modes to enhance scalability for massive clusters. The masquerading helps pods egress using the node IP. If you're using Cilium with IPAM in ENI mode, it allows you to assign VPC IP addresses directly to pods. So, it seems the NAT might not be necessary in this case, which aligns with your inquiry.

That's quite similar to what we did! We turned off egress NAT/masquerading and let the VPC NAT gateway take charge of the egress for the cluster. We’re running IPAM in ENI mode on EKS without the EKS kube-proxy. After switching to this configuration (without masquerading), we had to restart all our services and pods that needed outbound connections to the internet, like AWS API calls. Unfortunately, the clients didn’t manage to recreate those connections smoothly because of the TCP connections.

From what I gather, the VPC NAT gateway mainly deals with traffic going from the VPC to the internet, not necessarily the routing from pods to the VPC. Implementing masquerading at the node level makes pod traffic appear as if it's originating from the node's IP, ensuring return traffic routes correctly, which can help avoid weird routing issues in some setups. However, if you're using ENI mode with VPC IPs for your pods, many people actually disable it since the pods already have routable VPC addresses. It really hinges on your specific networking configuration.

There is indeed an interface for each pod's IP. You’ll want to identify the interface that’s not used for pod-to-pod routing but for external traffic instead. That’s crucial in managing how your network traffic is handled.