I use tools like Prometheus, Tempo, Loki and Elasticsearch, but haven't worked with OpenTelemetry or its Collector. Is it worth running a Collector in every cluster, or are there cases where the existing setup is enough?
When does OpenTelemetry add value if I already have Prometheus and other observability tools?
Answers
OpenTelemetry is more than the Collector. It includes APIs, instrumentation libraries, conventions and a protocol for producing and exchanging telemetry across languages. The appeal is that your application instrumentation doesn't have to be tied as closely to one storage or visualization vendor.
That complements tools like Tempo or Elasticsearch rather than replacing them. The Collector is an optional component that can receive, process and forward data to those backends.
Reply to this answer
For a cluster where Prometheus already scrapes the metrics you need, I wouldn't add a Collector just to put another hop in the path. Prometheus scraping a Collector isn't automatically better than scraping the applications directly.
Consider it when you have a concrete need, like sending the same metrics to more than one destination or applying processing before export. Otherwise, the extra component is another thing to configure and operate.
Reply to this answer
Tracing is a good reason to look at OpenTelemetry even if your metrics setup is settled. The instrumentation libraries and common conventions can help services written in different languages produce traces for a backend such as Tempo.
Check what your spans contain before sending them anywhere. Request data and other sensitive information can end up in traces, so privacy and retention deserve some thought.
Reply to this answer
The Collector can be useful as a shared collection pattern. It can scrape Prometheus endpoints and receive other telemetry, then forward data to the destinations you configure. That can simplify routing when a team needs to send metrics to two backends, for example.
Grafana Alloy is another option built around the OpenTelemetry Collector ecosystem. It can work with Kubernetes service and pod monitor configurations, although whether it replaces an existing Prometheus setup depends on what that setup does.
Reply to this answer
Add your answer
Answers are reviewed before publication.
4c~ferrylatch_926:
So I could use OpenTelemetry instrumentation without putting a Collector in the cluster?
_g34~torkavi.8:
Yes. Instrumented applications can send telemetry directly to a compatible backend. A Collector adds a processing and routing layer when you need one.