Why Is GPT-5.6 So Slow in Azure Foundry, and Is Prompt Caching Working?
Asked by
MellowPine47
GPT-5.6 in Azure Foundry has been extremely slow for me, especially when using the Responses API. Time to first token can be around a minute, making the model nearly unusable, while output speed afterward seems normal. Prompt caching also appears not to work reliably, even though several programming features require this API. Has anyone figured out whether the problem is a caching bug, Azure capacity constraints, or something else? Is a proper fix or redeployment expected soon?
Answers
Be careful when evaluating costs and caching. Unexpectedly large charges can result from selecting provisioned throughput instead of a standard serverless deployment, so check the deployment type and billing configuration before assuming the model has no caching. Also monitor usage closely while the service is unstable.
Reply to this answer
Prompt caching may be affected by an upstream API bug rather than by the model weights. However, Azure hosts and serves its own deployments, so the Azure and direct API implementations can behave differently. The fact that caching works through the direct endpoint but not consistently in Azure suggests the Azure Responses API integration still needs attention.
Reply to this answer
Some users are reporting that performance has improved recently, possibly because caching is now being applied more consistently. It may be worth testing again, but caching behavior still needs to be verified separately for implicit and explicit cache modes.
BrightClover7: Has anyone confirmed that both implicit and explicit caching are actually working in Azure, rather than just seeing faster responses from general capacity improvements?
Reply to this answer
The latency looks more like Azure capacity saturation or request queueing than a problem with the model itself. Other hosted endpoints show normal generation throughput but unusually high initial latency, which suggests limited inference concurrency. Capacity may improve as more resources are brought online.
VelvetMango2: That would fit the difference I’m seeing: roughly 64 seconds to first token through Azure versus about 1.2 seconds through the provider’s direct API, while the output rate is otherwise similar.
Reply to this answer
Add your answer
Answers are reviewed before publication.
LunarKite39: Exactly—the same request works with the direct API, including both implicit and explicit caching, so it’s difficult to treat this as only an upstream issue.