When a Kubernetes workload breaks, do you normally begin at the cluster and node level and work downward, or start with the failing pod and expand the investigation from there? I'm trying to build a consistent troubleshooting workflow instead of running unrelated kubectl commands until something looks suspicious.
3 Answers
Start by measuring the blast radius. One pod failing is a very different situation from many workloads failing across several nodes. Check whether the issue affects one workload or many, one namespace or multiple, and one node or multiple nodes. Also ask what changed recently. Once the scope is clear, follow the symptom: investigate the pod for a CrashLoopBackOff, but look at nodes or cluster services when unrelated workloads are affected. The key is to form a hypothesis, gather evidence, and narrow the possibilities instead of following a fixed command checklist.
For a single failure, I usually go symptom-first: check the pod status, recent events, its description, container logs, exit code, probes, configuration, mounts, and dependencies. If the pod looks healthy, then expand to the service, networking, storage, or external dependencies. A quick look at node capacity can also reveal that the apparent pod problem is really resource pressure.
A practical order is events, describe, logs, then zoom out. Sorting recent events by timestamp can quickly reveal scheduling failures, image-pull errors, or node pressure. If several unrelated pods fail together, switch to investigating the node or cluster dependencies such as DNS, the CNI, storage, the API server, admission webhooks, or cloud-provider services. The right starting point depends more on the failure’s scope than on a strict pod-first or cluster-first rule.
The pod needs to be checked first so you know whether it was scheduled and started at all. Events and a description often distinguish scheduling, image-pull, probe, and node-pressure problems before logs become useful.