We're running several microservices with OpenTelemetry and Tempo, but reduced tracing to about 5% to keep costs under control. What's your usual workflow when you need a complete trace for a specific request that wasn't sampled? Do you temporarily raise sampling to 100% and retry, use a debug header or another forced-tracing mechanism, rely on correlated logs, or handle it some other way? I'm especially interested in how teams investigate successful requests with issues like stale data, since error-only sampling won't catch those.
2 Answers
For on-demand debugging, a forced-tracing header tends to scale better than changing the global sampling rate. We use a controlled debug flag, such as a force-trace header, to make one request and its downstream calls sample at 100%. The trace ID is also included in logs, so you can follow the request even when neither normal sampling nor the debug path captures a full trace. Tail sampling remains useful for errors and slow requests, but it can’t recover a trace that has already been dropped.
Tail sampling is useful here: keep a small percentage of normal successful traces, while retaining essentially all traces that end in errors or meet conditions such as high latency. That gives you much better coverage for failures without storing everything.
That helps with errors, but it doesn’t solve cases where the request succeeds and still has a problem, such as returning stale data. Tail sampling may discard those traces before you know they matter.
Is the forced sampler implemented separately in each service’s SDK, or can it be configured centrally in the collector? I’d prefer an off-the-shelf option rather than maintaining custom logic in every service.