Hey everyone, I'm currently involved in a project where my goal is to enhance the observability of our services. I want to help our developers understand what's happening with their code and assist the ops team in tracking CI activities effectively. While I could opt for a paid solution like Datadog, I'm interested in exploring local, open-source alternatives.
In my experience, I've tried a few tools like OpenSearch with dataprepper and Grafana combined with Alloy, Tempo, Loki, and Prometheus. However, I've noticed that whenever something goes wrong, developers often struggle to pinpoint which microservice in our observability stack is failing, and some aren't even aware there's an issue.
Recently, I've come across a lightweight setup involving davetron5000/otel-desktop-viewer along with Prometheus and Dozzle for logs. It's quite simple and doesn't require complex configurations, but perhaps lacks some correlation features.
Given this, I'm curious about what toolset you all use in similar scenarios. What works best for you?
4 Answers
That sounds interesting! I’m curious about how you're getting insights into CI with that setup. Are you using something like the GitHub receiver from OpenTelemetry?
Also, I wanted to mention that we're developing SigNoz, an open-source observability platform that integrates metrics, logs, and traces. We've recently implemented CI/CD observability features which help track repository and pipeline health metrics, like average merge times and flaky pipelines. It might be worth checking out!
Have you considered using eBPF combined with Groundcover? It automatically captures metrics, logs, and traces without requiring any code changes, which is super handy for developers who need quick insights.
I hadn't heard of Groundcover before, but it sounds promising! I’ll look into it, thanks!
Great question! I've been on the lookout for similar setups myself. I think Sentry Spotlight has some capabilities that align with what you're looking for.
For local development, I've been using Aspiredashboard. It's been quite effective for my needs.
Thanks for sharing that, it looks interesting!
For CI, we use an otelcol configured correctly with the right credentials during PRs tagged with `debug`. I'll definitely check out SigNoz; it sounds like what I’m looking for!