We rolled back a prompt and saw staging recover, but three serverless regions were still serving different instructions. One used the code default, another read an environment variable, and a third served a stale edge-cache value from the previous deployment. An unnoticed model revision introduced another variable, making our experiment results unreliable. We also lacked lineage connecting an immutable prompt version to the region, cache state, model metadata, and request timestamp. What should a rollback actually prove unless every trace includes those fields and environment parity is checked first? How are you enforcing immutable prompt versions across regions, and do you fail closed when model or cache metadata is missing?
4 Answers
Prompt lineage is the missing control here. Store immutable prompt versions and require every application instance to reference one explicitly rather than falling back to defaults or loosely managed environment variables. I’d also include an application-level version hash so rollback validation doesn’t depend on a single dashboard.
Add a small endpoint in every region that reports the active prompt version, model revision, configuration source, and cache state. Run it as an automated parity check before and after deployment, and block the rollout if regions disagree.
Make the observability fields mandatory on every request trace: immutable prompt ID or hash, model revision, region, configuration source, cache hit or miss, and request time. If those fields are optional, they’ll disappear exactly when you need them for an incident.
That’s the main lesson for us too. Optional metadata wasn’t present when we needed to reconstruct the rollback.
A rollback is only conclusive if you can prove what was actually served. Treat missing model, prompt, region, or cache metadata as an error for production analysis—and, where practical, fail closed or route traffic away from instances that can’t report a complete serving identity.
Immutable versions don’t help much unless every region is forced to reference the same version and stale cache entries are explicitly invalidated.
Prompt ID and cache state would have saved us the most investigation time. The application hash also sounds useful for detecting hidden configuration drift.