I'm running the Mosquitto MQTT broker on a single-node Talos cluster with Cilium. I've set up the service as a LoadBalancer, and it's getting a VIP that's advertised via BGP. Incoming traffic hits the pod with the correct source IP, but outgoing traffic uses the node's IP instead. This causes connection issues because the external MQTT broker can't recognize responses from a different IP, leading to timeouts.
I've tried a couple of things:
1. I disabled SNAT in Cilium using ipMasqAgent, which helped a bit, but now the outgoing traffic shows the pod's in-cluster IP as the source rather than the Load Balancer VIP.
2. I attempted to set up an egressGateway, but couldn't get it to work as it seems to require a node with the egressGateway as its IP.
I'm looking for ideas on how to make sure that traffic sent in response to LB traffic originates from the LB VIP instead. Any suggestions?
4 Answers
It seems like you're dealing with some IP masquerading issues. Check out Cilium's documentation on masquerading. It might give you insights into the configuration specifics you need to tweak to get this working.
Here's a link to the docs: https://docs.cilium.io/en/stable/network/concepts/masquerading/
Honestly, Cilium doesn’t give you that option. To do what you want, an egress gateway is necessary, but it will have a different IP than the Load Balancer's VIP.
But if that’s the case, the issue still persists, just with another IP. I find it surprising that this isn’t a more common problem! Is it just the Mosquitto broker being fussy about the source IP for TCP replies?
You might be mixing up two scenarios: 1) Outbound traffic initiated from outside the cluster, and 2) Outbound traffic initiated internally. If there's no misconfiguration, the first scenario should just work. However, for the second, Cilium doesn’t support this out of the box. I think the enterprise version has features like BGP advertised egress gateways that might cover this.
Unfortunately, egress gateways with a VIP setup isn't feasible—it’s an enterprise feature.
I don't believe this is just a misconfiguration. The documentation states that Cilium will automatically masquerade the source IP of all outgoing traffic to the node's IP, which is routable. You're looking to masquerade to the LB's IP, and that option doesn't seem to exist. It might be limiting.