I'm trying to figure out a reliable way to ensure that a Kubernetes pod is not only running but that all its containers have been up for a minimum period. I've noticed that the pod's status can be misleading—just because it shows as running, it doesn't mean that all containers are healthy. For example, if a container restarts due to a failed liveness probe, the pod's start time won't change, which makes it hard to track how long the containers have actually been running.
I've come up with a method to check this using kubectl and yq. Here's a snippet of the command I use:
```bash
timestamp=$(KUBECONFIG=$wl_kubeconfig kubectl get pod -n kube-system -l app.kubernetes.io/name=cilium-operator -o yaml | yq '.items[].status.conditions[] | select(.type == "ContainersReady") | .lastTransitionTime' | sort | head -1)
```
However, I'm looking for suggestions on how to improve this. My goal is to ensure that all containers in a pod are healthy for at least 120 seconds before proceeding with tests, especially since I've seen cases where a pod shows ready status only to face issues shortly after. Any better solutions or tools would be greatly appreciated!
Answers
Have you considered using `minReadySeconds` for your deployment? It allows you to specify the minimum time a pod must be ready before it's considered available. This could help you ensure that your containers are indeed stable for a certain period.
Add a reply to this answer
It sounds like you're trying to differentiate between whether a pod is running and how long it's been running successfully. To check if a pod is running, you can look at the readiness and liveness probes. If both are fine, then it’s good! But for tracking uptime, why not use metrics from a monitoring solution? Maybe set up something like kube-prometheus? It'll give you detailed insights including uptime and restarts right in Grafana.
Replies
DevGuru88: I get that you're looking for something lightweight for CI. But a monitoring tool really simplifies your life in tracking these metrics effectively.
Add a reply to this answer
Add your answer
Answers are reviewed before they appear publicly.