When users start getting 503s, do you work from ingress through the service and pods, or start somewhere else? What are your first few checks, and which parts get repetitive?
Where do you start when users get 503s in Kubernetes?
Answers
First I'd figure out the scope. Is it one service, one pod, or several services? Check the alerts and service dashboards, then compare them with ingress metrics. If lots of services are affected, start higher in the request path. If it's isolated, go straight to that service.
Checks from inside and outside the cluster can help narrow down where requests stop working. If the external check fails but the internal one succeeds, look more closely at the path into the cluster.
Reply to this answer
Check what changed before walking every layer. A recent deployment or a readiness probe that starts failing under load can give you a much faster lead.
Then check the service endpoints. If there are no ready endpoints, inspect the pods, readiness checks, and service selector. That narrows the search, though it doesn't by itself prove there isn't also an ingress issue.
Reply to this answer
If the platform was working and this is a new problem, I'd start with the pods. Look for Pending, CrashLoopBackOff, or a bunch of restarts. Events and pod logs are useful once you have a suspect.
If the pods look clean, check the nodes too. A failing or newly provisioned node might give you another clue. From there, work back through endpoints and ingress as needed.
Reply to this answer
Don't forget traces if you have them. They can help show how far a request got and which service returned an error, especially when the visible 503 isn't coming from the service you first suspected.
Reply to this answer
The workflow also depends on who owns what. If you don't manage the cluster, gather the affected service, timing, alerts, and relevant logs, then bring in the team that does. That gives them somewhere to start without guessing.
Reply to this answer
Add your answer
Answers are reviewed before publication.
p.407+glint:
Readiness checks can be a repetitive one. A changed health check path or other config drift is easy to miss.