I'm experiencing a strange issue with my RKE2 cluster where outgoing traffic on ports 80 and 443 is always getting redirected to the nginx-ingress. Disabling nginx-ingress resolves this problem, but I'm curious about why this is happening. For instance, when I run a curl command from a pod to https://google.com, the output indicates that traffic is being intercepted by nginx-ingress and is returning a '404 Not Found' error. Here's the curl output for reference. I've checked my Cilium Helm config and haven't made any changes to CoreDNS. I've also just deactivated some features like L7Proxy and GatewayAPI that I had activated before testing. Any insights into this issue or things I might be missing?
6 Answers
It might be that your nodes have a local DNS search list set up that includes a wildcard for your ingress. I ran into a similar situation with CoreDNS where it affected the resolution due to the Ndots behavior. You might want to double-check your DNS settings to rule this out.
Have you made any modifications to your CoreDNS settings? Sometimes tweaks can cause unexpected behavior.
Nope, I haven't touched the CoreDNS settings. I'm still using the default RKE2 configuration.
It's worth using dig to see how your DNS is resolving. And could you let us know which versions of Cilium and nginx-ingress you’re currently using? The latest Cilium docs don't mention ciliumDNS, so that might be a clue.
You could try rebuilding your nginx-ingress controller with a proper ingressClass instead of using systemDefaultIngressClass. It sounds like your egress traffic is being hijacked by nginx-ingress. I did a quick glance, but I’m super tired. Let me know how it goes after you try this!
Could you provide your HelmChartConfig for Cilium? It might help others to identify the issue more effectively.
I've added the HelmChartConfig in my original post for review.
Was that an output typo in your curl result with both the IPv6 and IPv4 addresses listed? If it’s consistently showing 404 from the nginx-ingress, that’s a clear sign of misrouting somewhere. Check your ingress rules as well.
I've looked into the DNS configuration and it seems fine. When I run curl with the direct IP https://172.217.168.78, I also get a 404 from nginx-ingress. So I don't think it's a DNS issue.