Hi everyone! I'm having trouble with Ingress-NGINX while running a WebSocket-based service on Kubernetes under heavy load, and I could really use some advice to diagnose the problem.
Here's a bit about my environment: I'm working with a setup that includes a client connecting to HAProxy, which then routes traffic to Ingress-NGINX (configured as a NodePort service), leading to a backend WebSocket API. My Kubernetes cluster has three nodes, and I'm using Ingress-NGINX installed via Helm in version 4.13.2, without any specific CPU or memory limits for the Ingress controller. When I run load tests with over 1000 concurrent WebSocket connections, Ingress-NGINX fails its health checks and eventually gets restarted by Kubernetes. Although the backend service tests healthy and reachable on its own, the NGINX logs indicate connection failures.
I'm curious if anyone has faced similar issues with Ingress-NGINX becoming unhealthy under heavy WebSocket loads. Could it be related to configuration limits, like worker connections or liveness probes? What are the best tuning settings for handling a large number of WebSocket connections? Any specific guidance for using WebSockets with Ingress-NGINX would also be appreciated! I've already successfully tested a similar setup on AWS EKS without this issue.
Thanks in advance for any help!
1 Answer
Hey, it would be useful to see your NGINX ConfigMap. Here are a few optimizations you could try: increase the `keep-alive-requests` to 10,000 and `upstream-keepalive-connections` to 2,000, as well as bumping `max-worker-connections` to 65,536. These changes can help manage high-concurrency scenarios better. Also, can you post your ingress resource spec? For more detailed recommendations, check the guide on WebSocket infrastructure in Kubernetes.
Thanks for the suggestions! I'll share my config as soon as I can.