Hey everyone! I'm trying to figure out how to send Kubernetes events as Slack messages. I've been exploring OpenTelemetry for gathering cluster metrics, and I kind of get that part. However, I'm stuck on how to send this information to a backend. I know Grafana isn't a data store, but I'll be configuring my alerts there. What should my workflow look like, and which tools should I be considering? I've found the OpenTelemetry documentation a bit lacking, and my searches often lead me to their SDK integrations instead of cluster metrics. I also posted a more detailed question on Stack Overflow.
Additionally, I'm curious about other potential solutions beyond the usual products like CloudWatch or New Relic. I've read that some folks have created custom solutions using Kubebuilder, which seems a bit complex for my needs. Thanks in advance for any help!
1 Answer
You might be missing a key piece here—something like Alertmanager can be really helpful for metrics. Check out this detailed guide on setting up Prometheus and Alertmanager with Slack: https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/
It's definitely not straightforward to promote Kubernetes events to Slack as first-class notifications. While coding it isn’t hard, there aren’t many good out-of-the-box solutions. You'll want to filter the events for relevance and manage duplicates, which is tricky. For example, Kubewatch is a tool, but it's a bit limited; however, kubestatewatch is a newer fork that might have what you need without the enterprise ties.