Hi everyone,
I've been using Prometheus through kube-prometheus-stack to monitor my Kubernetes clusters. My setup includes ServiceMonitor and PodMonitor CRDs that gather metrics from various components like kube-apiserver, kubelet, and CoreDNS, all beautifully visualized with default Grafana dashboards. I also integrated Loki and Mimir, with data stored in S3.
I'm considering replacing kube-prometheus-stack with Alloy for a more unified solution that manages both logs and metrics. However, I noticed that the k8s-monitoring-helm setup doesn't seem to support scraping for crucial control-plane metrics once Prometheus is removed.
I'm curious:
1. Why doesn't k8s-monitoring-helm scrape control-plane components like the API server, CoreDNS, and kubelet?
2. Do you add these endpoints to Alloy manually, or is there a way to reuse the existing CRDs from kube-prometheus-stack?
3. How have others approached this transition? What's the common practice in the industry when moving from Prometheus Operator to Alloy?
I'd love to hear your experiences, especially if you have Alloy running in production!
3 Answers
I had a similar issue. I just deployed Alloy next to kube-prometheus-stack and Loki, and it all went smoothly! It requires some configuration, but it's manageable. Just keep that in mind!
For my setup, I’m using Prometheus installed via the kube-prometheus-stack, but with Grafana disabled. I have a 24-hour retention policy and write to Mimir for long-term storage. I also collect logs with Alloy and Loki, and I'm utilizing Temp for traces. Each component has its own chart, which gives me flexibility to customize. All-in-one setups can be limiting for me.
I’m curious about why anyone would prefer Alloy. I find that Prometheus for metrics and Loki for logs work just fine together.
I’ve got a bare-metal setup without shared storage, and my 30-day Prometheus pod is huge. I had to set up Mimir with replicas for HA and connect it to S3. My goal is to transition to Alloy for direct data writing to Mimir while phasing out Prometheus.
Alloy specializes in collecting logs and sending them to Loki, which can simplify the whole workflow.
I moved from promtail to Alloy since promtail is on its way out. I needed something maintained to suit my use of Loki and Grafana.
Alloy acts as the agent to capture and send data. It replaces promtail and can scrape Prometheus endpoints as well as integrate with OpenTelemetry. Data storage’s still handled by Prometheus and Loki or their alternatives.