During an EKS upgrade, a kube-proxy node selector change removed it from most nodes and caused DNS failures. What would you monitor to catch that kind of cluster health problem sooner?
How do you monitor EKS for failures like kube-proxy disappearing?
Answers
I'd start with kube-state-metrics and Prometheus alerts for critical add-ons. For kube-proxy, compare the DaemonSet's ready count with its desired count, but don't stop there. A bad selector can reduce the desired count too, making desired versus ready look healthy while most nodes are missing the pod.
Also check kube-proxy coverage against the nodes where you expect it to run. Alert on CoreDNS deployment availability separately, since CoreDNS is normally a Deployment, not a DaemonSet. A DNS check from a workload gives you another signal when the pods look healthy but resolution isn't working.
Reply to this answer
Monitoring will tell you a release went wrong, but it won't prevent the change. Keep the add-on configuration in version control, review selector and toleration changes, and test upgrades in a preproduction cluster with comparable node labels.
Check that the system pods land on the expected nodes after the upgrade. A staging test can miss this if its node layout is too different from production.
Reply to this answer
For a starting stack, look at kube-prometheus-stack. It brings together Prometheus, Grafana, and Kubernetes monitoring components. You can build alerts around the add-ons you actually depend on rather than treating a dashboard full of green pods as a health check.
Reply to this answer
Keep an eye on node conditions and control plane metrics too. If you run Cluster Autoscaler, its events are worth monitoring for scaling trouble. Those signals cover a different set of failures from the add-on checks.
Reply to this answer
If you're already collecting telemetry with OpenTelemetry, the Collector is another way to build out cluster monitoring. Grafana-based and VictoriaMetrics-based stacks are options too. Pick something your team can operate and alert from, then test that the alerts fire for the failure modes you care about.
Reply to this answer
Add your answer
Answers are reviewed before publication.
r.416~quavrix:
Make the expected node set explicit in that coverage check. Some nodes may intentionally not run kube-proxy, depending on the cluster setup.