I ran six Terraform pipelines against one Azure subscription at the same time, all using the same service principal. Three started returning 429 errors within about four minutes. Each plan made roughly 1,800 read requests across around 300 resources, so the combined total was about 10,800 reads—below the subscription-level limit of 12,000 requests per hour per principal. However, Microsoft.Compute appeared to enforce a much tighter limit over a one-minute window, and each plan touched approximately 60 virtual machines. The burst exceeded that shorter window even though the hourly total looked safe. Retries then generated additional reads and increased pressure on the subscription-wide counter. I eventually divided the migration into four sequential phases, which kept the workload below both limits, but choosing the phase sizes was mostly guesswork. Azure exposes remaining quota through response headers on requests that have already been made, but there does not seem to be an endpoint for checking quota in advance. When multiple pipelines share throttling budgets but cannot see one another's usage, what is the best way to coordinate them without relying on a central counter?
4 Answers
The bigger issue may be the deployment structure. Split the estate into smaller Terraform states and workspaces, starting with separate landing zones and resource groups where appropriate. Smaller states reduce the number of reads per plan, limit blast radius, and make it easier to schedule independent pipelines instead of having every run scan the whole environment.
Separate service principals can help because some ARM limits are tracked by subscription, identity, and operation type. Giving independent workloads different identities may provide separate per-principal buckets. It is not a complete solution, though: subscription-wide limits and provider-specific limits such as Microsoft.Compute still apply. I would combine separate identities with lower Terraform parallelism and a cap on how many large runs can target the same subscription or region simultaneously.
Use the throttling information returned in ARM response headers as feedback, rather than treating the hourly subscription limit as the only constraint. A shared scheduler or pipeline queue can reduce concurrency when remaining capacity falls, while exponential backoff with jitter prevents all failed runs from retrying together. Azure’s subscription traffic metrics can also help you understand the aggregate request pattern, even though they do not replace coordination between active pipelines.
Terraform does not automatically coordinate separate executions that use the same credentials and subscription. Keep a bounded number of large runs active, reduce the provider parallelism setting, and serialize especially expensive VM operations. If possible, schedule work by subscription, region, or resource group so that the same Microsoft.Compute bucket is not hit by several plans at once.
If one repository or state file is managing nearly the entire Azure estate, that is probably driving the excessive request volume. Breaking it into smaller ownership and deployment units should help more than tuning retries alone.