How do you debug a request when its trace was sampled out?

November 16, 2026 4 answers
Asked by q.572+cloverhook

We have microservices using OpenTelemetry and Tempo with a low sampling rate. If a specific request wasn't captured, what's a practical way to investigate it or capture a useful retry?

Answers

_h24+driftcompass.6

If the trace was dropped and is no longer available, you can't recover it by changing the sampling rate afterward. Start with the request ID in your logs and follow it across services. That may give you enough to narrow down where things went wrong, although you won't get the trace view back.

q.572+cloverhook:

That makes sense for the original request. I'm also looking for a way to get a trace if the issue can be reproduced.

Reply to this answer
r.592+paperpiston

For a reproducible request, a controlled debug header can mark a retry for retention without turning up sampling globally. One option is to have ingress set a trace attribute and configure the collector's tail sampling processor to keep traces with that attribute.

The spans still need to reach the collector for that decision to work. Check any SDK head sampling and context propagation along the request path, or the collector may never see the full trace. Restrict who can trigger the debug path so arbitrary callers can't drive up telemetry volume.

_x82+signalcradle.3:

Does that require a custom sampler in every service?

Reply to this answer
VORLAX+7j

Tail sampling is useful when you can identify the interesting traces after the request runs. You could retain error traces and a smaller share of successful ones, for example, rather than treating both the same.

That won't catch every investigation, though. A stale data response might look successful to the sampler, so it could still be dropped unless you have an attribute or other signal that identifies it.

Reply to this answer
u.407~pocketvalve

If there's no targeted debug path yet, temporarily increasing sampling and retrying is a straightforward fallback. It affects more traffic than the one request, so keep an eye on telemetry volume and restore the previous rate afterward. It won't bring back the original trace.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions