We're running several microservices with OpenTelemetry and Tempo, but reduced head-based sampling to 5% to control observability costs. What's the best workflow when you need to investigate a specific request that wasn't captured? Do you temporarily raise sampling to 100% and retry it, use a debug header or trace-state flag to force sampling for that request, rely on tail sampling for errors and slow requests, or fall back to searching correlated logs? I'm especially interested in how teams handle non-error issues, such as stale data, that may not be retained by tail sampling.
2 Answers
A per-request forced-tracing mechanism is usually the most practical option. Add a controlled debug header or trace-state flag that tells the SDK to record and propagate the trace for that request, without changing the global sampling rate. Support or engineering can then reproduce the request and inspect the complete trace. Make sure the resulting logs include the trace ID so the trace and service logs stay linked.
Tail sampling is useful for retaining 100% of error traces and selectively keeping slow or otherwise interesting requests, while sampling normal successful traffic. The limitation is that it can’t recover a request that was already dropped, and it may still miss successful requests with problems like stale data.
Is this forced-sampling logic normally implemented separately in each service’s OpenTelemetry SDK, or can it be configured centrally in the collector? I’d prefer an existing component if one is available rather than maintaining custom code everywhere.