Why does an AWS NLB return ECONNREFUSED when ingress-nginx pods scale down?
Asked by
RHYTHM.in.5
We're seeing intermittent ECONNREFUSED errors when connecting to an AWS Network Load Balancer in front of ingress-nginx. The NLB uses instance mode, and the service is configured with externalTrafficPolicy: Local. The failures seem to coincide with ingress-nginx pods being restarted or scaled down. What causes this gap in traffic handling, and what's the best way to prevent it?
Answers
This is a common race between the NLB and externalTrafficPolicy: Local. When an ingress pod starts terminating, the node can stop accepting traffic immediately because there is no longer a local endpoint, while the NLB may continue sending requests to that node until its health checks detect the change. That short deregistration window can produce connection refusals. Add a preStop hook that sleeps for roughly 10–15 seconds so the pod has time to drain, and consider reducing the target group's deregistration delay from the default 300 seconds to around 30 seconds.
Reply to this answer
Readiness probes are still important, but they won't completely eliminate this problem on their own with an instance-mode NLB and externalTrafficPolicy: Local. The key issue is that the NLB's target registration state lags behind the pod and node state. Tighten the NLB health-check thresholds, configure startup and readiness probes for the ingress containers, and use a preStop sleep so the controller keeps serving while it is being removed from the load balancer. Also plan a longer-term migration away from ingress-nginx, but these changes should address the immediate failures.
RHYTHM.in.5: That makes sense. We'll add the probes and graceful termination settings while we work on the longer-term migration.
Reply to this answer
Add your answer
Answers are reviewed before publication.
RHYTHM.in.5: Thanks, that explains the timing. I'll test the preStop delay and a shorter deregistration period.