Our team is migrating a production retrieval-augmented generation agent from GPT-4o to GPT-5.1 in Azure AI Foundry because the GPT-4o deployment is nearing retirement. We kept the same system prompt, knowledge base, retrieval pipeline, and agent configuration, but GPT-5.1 behaves very differently. It produces longer, more heavily structured responses, frequently adds headings and summaries, and sometimes generates tables even though tables are explicitly disallowed unless the user asks for one. The answers are generally accurate and grounded, but the style is less appropriate for our users. Has anyone handled a similar production migration? We are considering prompt redesign, few-shot examples, reasoning or verbosity settings, output validation, a second rewriting pass, structured outputs, and regression tests based on previous GPT-4o responses. We do not need identical wording, but we would like to retain GPT-4o's concise, natural tone, avoid unnecessary tables, and keep responses grounded in retrieved content.
4 Answers
Check the reasoning-effort configuration. In one migration from a smaller GPT-4 model to GPT-5.1, lowering reasoning effort made the responses much more conversational and reduced the loss of the intended persona. It is worth testing several settings against a fixed evaluation set rather than assuming the default is suitable for a customer-facing RAG agent.
The quickest improvement is usually a revised system prompt that gives very explicit length and formatting rules. State the desired response size, tell the model to use plain paragraphs by default, prohibit headings and tables unless requested, and explain what to do when the retrieved context is insufficient. Longer term, create an evaluator using representative historical conversations so you can measure tone, concision, grounding, and formatting whenever you change the model, prompt, tools, or skills. That also helps control output-token costs.
Treat the migration like a software compatibility change instead of judging responses informally. Build an acceptance-test or end-to-end evaluation set from real but sanitized conversations, including short-answer cases, ambiguous questions, missing-context cases, grounding checks, and formatting rules. Score both factual quality and style. Then compare prompt versions, reasoning settings, and model variants using the same tests. This makes it much easier to decide whether a rewrite pass or validator is actually improving the experience.
You may want to compare a smaller GPT-5 model variant instead of moving directly to the full GPT-5.1 model. A mini variant can be closer to GPT-4o in response style while offering quality and context improvements, with fewer prompt changes and lower cost. An output validator is still useful for catching unwanted tables or formatting, but it should complement a good prompt rather than be the only control. For enterprise RAG systems, predictable behavior and a stable evaluation baseline can matter more than always selecting the newest model.
That is especially relevant when model availability depends on the deployment data zone. Some environments receive newer versions later, so the most practical migration target may be the one that is available consistently in the required region.
A useful evaluation suite should include concrete assertions such as maximum or typical response length, no table when one was not requested, citations or claims supported by retrieved passages, and an appropriate fallback when the answer is not in the knowledge base.