Hey everyone, I'm feeling pretty overwhelmed with my k3s cluster setup using Cilium and BGP, and I could really use some help. I've spent over five days trying to troubleshoot issues I'm having with asymmetric routing and hairpinning in my BGP configuration. Here's my current setup:
- My network is 10.10.1.0/24, with the router at 10.10.1.1.
- My nodes are infra1 through infra8 with addresses 10.10.1.11 to 10.10.1.18.
- I've set up a VIP (Virtual IP) at infra-apt with an address of 10.10.10.6.
- The service currently has externalTrafficPolicy set to Local, and for testing purposes, it's pinned to infra1 (10.10.1.11).
- I'm running Debian on infra1 (6.1.0-40-amd64) and Raspbian on the others.
I've shared my Cilium and BGP configuration files in the links provided, and here's what I've observed so far:
- I can access a simple web service inside the cluster through 10.10.10.6 from my k3s nodes directly.
- However, I can't access 10.10.10.6 from an external laptop on the same network.
- Interestingly, I can access the services using DNS like Pi-hole from the laptop, but not via direct IP.
- I can't ping or traceroute to 10.10.10.6 from anywhere unless I'm using TCP mode, which causes a routing loop.
I'm suspecting that the return traffic is not following the correct route. The only way I can access the TCP service at 10.10.10.6 is by adding a static route on my laptop or modifying the router's iptables rules, but that feels like a temporary fix.
It's really driving me crazy trying to get this working properly! Any insight or advice on how to debug and resolve this would be greatly appreciated! Thanks!
1 Answer
Hey! It sounds like you're dealing with some pretty standard asymmetric routing issues here. First off, have you checked if your k8s nodes have all the necessary forwarding settings enabled? A quick look at your router's routing table could help confirm that everything is in order too.
As for the lack of ping/traceroute responses, this is often expected for VIPs advertised outside the cluster. Given your setup, a laptop on the same LAN as the nodes might see the packets flow through the router and back in an unexpected way because of the different segments. A hairpin NAT rule on your router might help resolve this. Good luck!
Thanks for the tips! I think you're right; my laptop's IP is on the same segment as the nodes which must be leading to the confusion since it's not using the VIP for return traffic. I’ll dig into the forwarding rules!