Our team is migrating a production RAG agent from GPT-4o to GPT-5.1 in Azure AI Foundry because GPT-4o is nearing retirement. We kept the same system prompt, knowledge base, retrieval pipeline, and agent configuration, but GPT-5.1 behaves quite differently. It produces longer, more structured responses, frequently adds headings and summaries, and occasionally uses tables even though tables are explicitly disallowed unless the user requests one. The answers are generally accurate and grounded, but the style is less suitable for our users. Has anyone handled a similar production migration? We are considering a revised system prompt, few-shot examples, reasoning or verbosity settings, structured outputs, an output validator, a second rewriting pass, and a regression suite comparing results with previous GPT-4o responses. We do not need identical wording, but we would like to retain GPT-4o's concise, natural tone, avoid unnecessary tables, and keep responses grounded in retrieved context.
4 Answers
Build a proper evaluation harness before tuning too much. Treat it like an acceptance or end-to-end test suite: include real questions, expected grounding sources, acceptable answer characteristics, and checks for verbosity, tone, unwanted headings, and tables. A regression set based on previous outputs is useful, but score the behavior rather than requiring exact text matches. That gives you a repeatable way to compare models and configurations.
It may be worth testing a model that is closer in behavior and cost profile to GPT-4o instead of jumping directly to a larger reasoning-oriented model. A current small or mini variant may require fewer prompt changes while still improving quality and context handling. In a production RAG system, predictable behavior and known limits can matter more than always selecting the newest model. An output validator for table formatting and a few input guardrails can provide an additional safety net.
Model availability can depend on the Azure deployment region or data zone, so the newest option may not actually be available in every environment. That could explain why some teams are limited to an older model.
I had a similar result moving from a smaller fourth-generation model to the fifth-generation model. Lowering the reasoning effort made the responses noticeably more conversational and helped preserve the persona in the prompt. It is worth testing reasoning settings alongside prompt changes rather than assuming the model swap alone is responsible.
Start with the system prompt, since it is the fastest way to bring the output back toward the desired style. Be very explicit about maximum length, when headings are allowed, and that tables should never be used unless requested. This can also reduce output-token costs. For the longer term, create an evaluator using representative past conversations to score tone, concision, grounding, and formatting. Then you can test prompt, model, tool, and skill changes against the same baseline instead of judging responses informally.
What kinds of cases would you include in a practical evaluation suite?