Hey folks, I'm really struggling to configure BGP with K3s and Cilium, and it's driving me crazy! I've been stuck for over five days trying to resolve some issues that seem to stem from asymmetric routing or hairpinning. Here's my setup:
- My network is 10.10.1.0/24 with the router at 10.10.1.1.
- The nodes have IPs ranging from 10.10.1.11 to 10.10.1.18 (3 control, 5 worker nodes).
- The Virtual IP (VIP) is 10.10.10.6 and I currently have the service set to externalTrafficPolicy Local, but I also gave Cluster a shot.
- For debugging purposes, the VIP is pinned to infra1 at 10.10.1.11, which runs Debian 6.1.0-40-amd64, while the other nodes are running Raspbian.
I suspect there's a configuration issue because:
- I can access the service internally using 10.10.10.6 from the nodes, but not from external devices (like my laptop).
- Access to the VIP from my laptop works if I use DNS but fails for pings and traceroutes (in TCP mode, it creates a routing loop).
- The only way to connect my laptop to the VIP appears to involve adding static routes, which isn't a sustainable solution since IPs can change.
I've tried various configurations with Cilium but couldn't get the VIP on infra1, or find routes that bypass the router in the return path. Any advice on how to fix my setup would be greatly appreciated! Thanks!
P.S. I found a similar issue on GitHub that might relate to my situation.
1 Answer
It sounds like you might have some routing quirks going on. Double-check that all your Kubernetes nodes have the right forwarding settings. I'm not sure about where your laptop is; is it on the same LAN as your nodes? Since the VIP is in a different range, packets might get confused and return the wrong way. You could fix this by adding a hairpin NAT rule on your router to ensure proper return traffic.
Thanks for the suggestion! Yes, both my nodes and laptop are on the same LAN, but I think that’s where the confusion is coming from. The nodes may not be routing back correctly due to ARP issues. Maybe I should explore those NAT rules.