How should we score live traffic for production drift?

Start with a small random sample, then deliberately oversample places where drift is more likely: new intents, low-confidence outputs, and inputs that look unlike the known eval set. Split the sample by feature or route so a small degraded segment doesn't disappear inside the overall average.

Keep the sampling method visible when you report scores. An oversampled group is useful for finding problems, but its score alone shouldn't be treated as the overall production quality rate.

I'd separate coverage drift from quality drift. First ask whether live requests still resemble the cases in the eval set. Then look at response quality within the segments that have changed. A shift in traffic doesn't automatically mean the model got worse, but it can show where your evals are no longer representative.

Use live scoring for discovery before wiring it straight to alerts. Look for recurring patterns and quality trends rather than paging on individual low scores. Otherwise a noisy scorer can turn the channel into something everyone ignores.

Once you know which trends are actionable, decide who reviews them and what they should do when one crosses the threshold. That can be a review queue before it becomes a page.

Feed the useful failures back into CI. When a production response exposes a gap, turn that case into a regression test so the next change is checked against it too. The fixed suite can keep evolving instead of remaining a snapshot of what you thought to test earlier.