Hey everyone,
I need some help figuring out a way to keep my StatefulSet of web serving pods running smoothly without interruptions. My main issue seems to revolve around the NGINX Ingress controller, which is juggling load balancing and maintaining session affinity based on cookies. I believe these two objectives might be in conflict.
Currently, I'm using MetalLB in Layer 2 mode to assign a load balancer IP to my NGINX Ingress controller, which has TLS termination, a default backend, and routing rules pointing to my backend service. My Ingress annotations are set to enable cookie-based session affinity with a custom cookie and local external traffic policy.
When everything works, the client is routed to the right pod until they log in and the cookie is set. But problems arise when that serving pod is taken down. The client sees a closed WebSocket, and upon reloading, they get a Bad Gateway error because the Ingress still tries to send traffic to the old pod that no longer exists.
I want the Ingress to effectively recognize when a backend fails and reroute requests as needed without having to wait for polling mechanisms. Polling is really slow, and I believe there's a better way to handle this. I've thought about implementing application-specific endpoints that respond quicker than standard health checks, but this doesn't help if the Ingress is continuously polling.
I'm also open to other ingress solutions if necessary, and I've heard replacing my current setup with HAProxy nodes might work. I'd love to hear your thoughts or experiences with anything related to this problem. Let's figure this out together!
3 Answers
You might want to consider using a session storage solution like Redis, which can help with scaling your app. It's really efficient, and a lot of teams have found it beneficial when working with NGINX. By moving away from a StatefulSet, you can unlock a lot of potential for handling more traffic too. This way, you won't run into issues with session persistence even if a pod goes down.
One interesting approach you could check out is implementing event-driven tests within your cluster. By using tools like Testkube, you can set up tests that monitor session cookies or WebSocket behavior when pods restart. This allows you to catch issues before the Ingress has to reroute. Just a thought based on what I've seen others trying!
It seems like there may be a potential enhancement for Kubernetes to speed up failover. Have you considered a change to how liveliness and health checks are executed? Instead of immediate responses, maybe there could be a mode where responses are delayed until the timeout approaches. It could improve fail detection. However, I’m not sure how to formally propose this. Any insights on where to start would be appreciated!
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically