Our team has relied on offline evaluations in CI for a while, but recently production inputs drifted and users started asking questions our test set did not cover. Quality declined in that area for weeks without any CI failures, which exposed the limits of evaluating only against a fixed dataset.
I'm now looking at ways to sample and score real production responses. Should we evaluate every request or only a percentage, and is it better to sample by feature or route rather than randomly across all traffic? I also want to detect meaningful quality drops without creating another noisy alert channel. How do teams decide what is worth alerting on, and who owns the response when an alert fires?
5 Answers
Track whether production traffic still resembles the data covered by your existing evaluations. A change in coverage or request distribution can look like quality drift, so it helps to measure input and feature drift alongside live response scores.
I’d start with targeted sampling by feature, route, or use case instead of one global random sample. That makes it easier to notice quality problems in lower-volume areas that might disappear in an aggregate score.
Begin live scoring as an exploratory and measurement phase, not an immediate paging system. Learn which traffic patterns, segments, and metrics actually correlate with user-visible problems before attaching alerts. Otherwise the team may get flooded with false positives and start ignoring the channel.
That also gives you time to establish useful thresholds and baselines. Alerting too early can create noise before you understand normal variation.
Treat production traffic as another source of evaluation data rather than keeping it separate from CI. When you find a poor response in live traffic, capture it, review what went wrong, and turn it into a regression case. That lets the evaluation suite evolve with the product instead of staying frozen around assumptions from launch.
The basic loop is: discover the issue in production, then lock it into CI so the same failure is less likely to return.
Put deployments, prompt changes, and live quality scores on the same timeline. A sudden step change after a release is much easier to investigate when you can see exactly what shipped. For ownership, the team that made the relevant change should generally own the initial investigation and follow-up.
That distinction seems important: first determine whether the evaluation set stopped representing reality, then decide whether the model or prompt quality actually degraded.