I've been facing some irregular 504 Gateway Timeout errors in our Kubernetes setup on GKE. Initially, I thought the app was the culprit, but after checking the logs, everything seemed fine. It turns out the issue was with the load balancer's timeout settings rather than the application itself. We managed to fix it on the infrastructure side using BackendConfig. I'm curious about how others typically troubleshoot 504 errors in Kubernetes. Do you usually start your investigation from the ingress/load balancer side or do you look into the application layer first?
2 Answers
I always check the ingress first. Most of the time, the load balancer settings are where the timeout issues start. If that checks out, then I look at the application.
I documented the whole debugging process and fix if anyone wants to take a look: [Here's what worked for us in fixing 504s in GKE](https://www.kubeblogs.com/fixing-504-errors-in-gke-load-balancer-how-backendconfig-solved-our-30-second-timeout-problem/). Just a heads up though; if your API is taking too long to respond, it might be more of a design issue than just a timeout problem.
I get that point, but still, APIs should ideally respond quickly. If you have tasks taking longer, consider using async responses to avoid connection drops.