I'm trying to figure out what exactly happens to application traffic when a worker node unexpectedly goes down in my Kubernetes cluster. The setup is a deployment with two replicas and affinity rules to ensure the pods run on separate nodes. I also have a LoadBalancer service that matches these pods, with an external IP managed by MetalLB announced via BGP. When a node crashes, my understanding is that there's a lag during which Kubernetes still thinks both pods are ready, causing traffic to be sent to the dead pod for about 50 seconds. After this period, the node gets marked as 'NotReady', and traffic then routes only to the live pod. I wonder, how do others handle this situation? Are there Kubernetes settings or features that can minimize traffic loss in this scenario? Or is it necessary to implement something like an F5 Load Balancer in front of Kubernetes?
2 Answers
Absolutely agree with focusing on how MetalLB is set up. The local traffic policy is a game-changer! Also, if your application can afford some transient traffic loss, a proper L7 proxy will handle health checks, which improves response time during outages. Just analyze your failure tolerance as that determines whether you want a dedicated load balancer solution like F5 in front to manage these situations better.
If you’re using MetalLB, you should set the `externalTrafficPolicy` to `Local`. This way, MetalLB will only announce the LoadBalancer IP on the nodes that have a 'Ready' endpoint for the service. Once the node crashes, the BGP speaker on that node stops advertising, and no other nodes without an endpoint will advertise it either, which should direct all traffic to the healthy pod. You can also tweak the `node-monitor-grace-period` and `pod-eviction-timeout`, but be cautious as it might increase load on your control plane. Additionally, look into configuring IPVS for faster service entry removal when using kube-proxy, and consider a good L7 ingress proxy for health checks, which can minimize the recovery time.
Such a solid response! It’s great to see such detailed advice shared here.
Thanks for explaining the `externalTrafficPolicy`. I didn’t know that setting could help with routing traffic better! You’ve given me a lot to consider.
Thanks for your insight! I think improving response time for failures is key for us, so I’ll definitely explore using a dedicated solution.