I'm running an LGTM stack on k3s, and Grafana is currently the only component working. Everything was fine until yesterday, without any intentional configuration changes. The Prometheus service still exists at the expected address, but the endpoint cannot be reached. After investigating, the Prometheus pod began failing its health check and entering CrashLoopBackOff because it wasn't responding before the probe deadline. What should I check to identify the underlying issue?
4 Answers
Start with the basics: verify that the Prometheus pods are actually running, inspect their current and previous logs, and check whether they are in CrashLoopBackOff. Also look at CPU and memory usage, available disk space, and the pod events. A service can exist and still be unreachable if its endpoints are unhealthy or the backing pods are repeatedly restarting.
The health-check timeout may be a symptom rather than the cause. Prometheus can become so resource-hungry that it stops responding to probes in time. Check its memory consumption and the node's overall pressure. In my case, Prometheus exhausted the node's RAM and caused the entire node to fail. Increasing resources, reducing retention, or fixing the storage and ingestion load helped.
The persistent volume was the main issue in my case. I had left the Prometheus PVC at its default size, and the volume filled up quickly. After increasing the volume size, Prometheus recovered. Check the PVC and node filesystem usage, and make sure retention and log rotation are configured so old metrics and logs don't consume all the available space.
Also consider the physical environment if this is running on local hardware. Excessive heat can cause thermal throttling and make the whole cluster unstable, which can look like a Prometheus networking or probe problem. Check node health, temperatures, and system events alongside the Kubernetes diagnostics.
That turned out to be part of my problem too—the PVC was much smaller than intended and had completely filled up.