I ran six Terraform pipelines against one Azure subscription at the same time, all using the same service principal. Three began returning 429 errors within about four minutes. Each plan made roughly 1,800 read requests across 300 resources, so six plans produced around 10,800 reads—below the subscription-level limit of 12,000 requests per hour per principal.
The problem appears to be a shorter Microsoft.Compute limit. Each plan touched about 60 virtual machines, creating a burst that exceeded a per-minute provider limit even though the hourly total looked safe. Retries made things worse by generating another wave of requests and pushing the subscription-wide counter closer to its limit.
I eventually divided the migration into four sequential phases, which kept the request rate below both limits, but choosing phase sizes was mostly guesswork. Azure includes remaining-quota information in some response headers, but there is no separate endpoint for checking shared usage before making requests.
When several pipelines share throttling budgets but cannot see one another's consumption, what is the best way to coordinate them without relying on a central counter?
4 Answers
The bigger issue may be the deployment structure. If everything is managed from one large repository and state file, split it into smaller states and landing zones first. Separating resources across subscriptions where the architecture allows it will reduce shared throttle pressure and make failures less disruptive. A large environment generally benefits from multiple subscriptions rather than one subscription containing the entire estate.
There are subscription traffic and usage metrics that can help you observe overall activity, so check those alongside the response headers. The headers can report remaining request capacity after calls are made, but they are useful feedback rather than a reliable preflight quota query.
Separate service principals may help because some ARM limits are tracked by subscription, principal, and operation type. Giving independent workloads different identities can provide separate buckets, although it is not a complete solution: subscription-wide limits and Microsoft.Compute throttling still apply. I’d combine that with lower Terraform parallelism and a limit on how many large runs can target the same subscription or region at once.
Terraform does not coordinate throttling across separate concurrent runs. Reduce the per-run parallelism, stagger the larger plans, and use a shared pipeline scheduler or lock when several jobs target the same subscription and provider. Retries should also use exponential backoff with jitter so all failed runs do not immediately create another synchronized burst.
If this is effectively one monolithic deployment for the whole Azure estate, splitting the repository and state files should probably come before fine-tuning request limits.