I'm running EKS with Karpenter and would like a single dashboard showing which node was consolidated into which replacement node, along with the pods affected and where they were scheduled afterward. I use Datadog, but the Karpenter disruption logs generally show the source and destination nodes plus a pod count, while Kubernetes events provide pod-level eviction and scheduling details. The challenge is that Deployments usually recreate pods with different names, making it difficult to connect the original pod to its replacement. Can this be built reliably in Datadog, or are there better tools or approaches? I'd especially appreciate examples from anyone who has implemented this hands-on.
5 Answers
A practical approach is to combine Karpenter disruption events with pod eviction and scheduling events. Use the disrupted node, replacement node, and a tight time window as join keys. The consolidation event may include the evicted pod names and target node, while scheduling events show where the replacement pods landed. Matching by Deployment or ReplicaSet ownership makes this more reliable.
There are also node-focused tools and emerging Karpenter interfaces that show node creation, draining, removal, capacity, cost, and pod placement. They may not reconstruct the full old-pod-to-new-pod relationship, but they can give a clearer operational view than raw events alone. An event-relay workflow that sends Karpenter and scheduling events to the same notification system is another useful option.
Pods usually aren’t moved in place: the old pod is terminated and a replacement is scheduled. For StatefulSets, stable names make the relationship easier to follow. Deployments are harder, so correlate using the owner references, such as the Deployment or ReplicaSet, rather than pod names alone.
Datadog can handle much of this. The built-in Karpenter dashboard is a useful starting point, and adding Karpenter logs plus Kubernetes events provides better context. You’ll probably need to parse the relevant fields and build a custom correlation view rather than expecting a direct pod-to-pod mapping.
It may be more useful to focus on application health and unexplained restarts than on treating every pod replacement as a failure. Still, a restart and disruption dashboard can help identify services that lack adequate disruption handling, OOM kills, or repeated manual deletions. Filtering out expected consolidation-related restarts can make the remaining signals much more actionable.
That makes sense. I’m looking at owner references and pod UIDs now, since matching only by names breaks whenever a ReplicaSet creates a replacement.