I'm running the Mosquitto MQTT broker on a single-node Talos cluster with Cilium and I'm facing an issue. I have exposed the service as a LoadBalancer, and I can see the VIP being advertised via BGP. While incoming traffic reaches the pod with the correct source IP from outside the cluster, the outgoing traffic is originating from the node's IP instead of the LoadBalancer VIP. This is causing MQTT connection issues because the external broker doesn't recognize replies coming from the node's IP, which leads to timeouts. I want to know how I can make sure that outgoing traffic in reply to what arrives at the LoadBalancer is sent with the LoadBalancer VIP as the source address. I've already tried disabling SNAT in Cilium, which just changes the outgoing traffic to use the in-cluster pod IP instead of the LoadBalancer VIP. I also attempted to use an egress gateway, but that didn't work as expected. Any suggestions on how to resolve this?
1 Answer
It sounds like you might be dealing with a configuration issue related to IP masquerading. You should check the Cilium documentation on IP masquerading to ensure everything is set up correctly. Sometimes these masquerading settings can mess with how source IPs behave for different services.
I get what you're saying, but the documentation states that Cilium will automatically masquerade the source IP of traffic leaving the cluster to the node's IP since it's routable. You want to change that to the LoadBalancer's address instead, which doesn't seem to be an option.