Hey folks, I'm new to Kubernetes and trying to figure out how to monitor my EKS cluster effectively. I'm leaning towards using the OTLP protocol to push metrics and logs from my applications. Ideally, I want to avoid using agents or collectors like Alloy or the otlp-collector. However, I realize that might mean missing out on pod logs, but I'm okay with that since I'll be pushing logs anyway.
Currently, I'm focused on gathering node and pod metrics, which means I'd need to deploy Prometheus and Grafana along with adding Prometheus scrapers. But I'm a bit overwhelmed by all the different deployment methods out there—like the Prometheus operator, kube-prometheus, and various Grafana charts—each doing similar things yet in slightly different ways. I'm also unsure how compatible these tools are with each other. So, how did the observability space get so complicated? Any tips on how to approach this?
3 Answers
There’s actually a Helm chart you can use to set everything up, and yes, you can avoid those agents if you like—but do keep in mind you should have some form of monitoring within your cluster.
Also, if you're setting up alerts based on cluster health, consider running your monitoring on a separate cluster. It’s like putting the fire alarm in your electronics—the alarm won’t help if the system is down! Trust me, nobody wants to find they can’t access their monitoring because it went down along with their main cluster.
Also, be careful about the volume of data you’re collecting; logs can grow out of control quickly! Look into `chronological compactors` like Thanos for Prometheus/Loki to manage storage effectively.
While I won't tackle your main question directly, I’d point out that `kube-prometheus` is a Helm chart that deploys the Prometheus operator along with all required CRDs. This operator manages the deployment of Prometheus itself.
On the other hand, `kube-prometheus-stack` bundles together multiple Helm charts, including `kube-prometheus` and generally Grafana by default. I suggest taking the time to read through the Helm charts to grasp what each component does; understanding this can clear up some confusion. Most setups you'll see revolve around Grafana with either the LGTM stack or a combination of Prometheus, Grafana, Fluentd (or Fluentbit), and tracing tools like Jaeger.
Honestly, don't stress too much about it! If you're open to it, I really recommend using Alloy or the OTLP collector with the LGTM stack—Loki for logs, Prometheus for metrics, and Tempo for traces. Just have all your containers use their OTLP SDKs.
To get started with the LGTM setup, you can use the k8s-monitoring-helm chart from Grafana; it’s super straightforward! Trust me, once you start scaling up, you’ll look into operators and other options later on.
Related Questions
How To Get Your Domain Unblocked From Facebook
How To Find A String In a Directory of Files Using Linux