How Can I Monitor When a Pod Is Killed After the Grace Period?
Asked by
CuriousPanda33
Hey everyone,
I've got some worker pods that occasionally run for a long time, and I've set a termination grace period for them. I'm wondering if there's a straightforward way to monitor when a pod is actually killed because it exceeded that termination grace period. I'm looking to set up a monitor in Datadog for this issue. From what I understand, kubelet may not send a specific event for this scenario.
Thanks a ton for your help!
Answers
Exactly! The termination grace period only comes into play when the pod is shutting down, so if it runs out and the pod is still active, it gets forcefully killed. This can definitely be a scenario worth tracking to avoid data loss. I'm also interested in finding a solution for this!
Reply to this answer
It sounds like you're dealing with some pods being killed before they finish their tasks, which might mean your termination grace period is set too low. You definitely want to monitor this situation to ensure your workers are being given enough time to shut down gracefully. Have you considered using a tool like Grafana to set up some monitoring for that? It can really help you get the insights you need.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions