An event stream flooded a pipeline, and downstream requests hit an ERP rate limit. Azure WebJobs and Databricks jobs failed as a result. What would you change to keep a burst like that from cascading through the workflow?
How do you stop ERP rate limits from taking down data jobs?
Answers
Start by controlling how quickly work reaches the ERP, not just retrying after it gets rejected. A queue or other buffer can absorb a burst while consumers send requests at a rate the ERP can handle.
For throttled requests, use exponential backoff with jitter and a retry limit. A circuit breaker can pause calls when the integration is repeatedly failing, but make sure the paused work has somewhere to wait rather than disappearing.
Reply to this answer
Measure the burst rate from the event stream alongside the ERP limit. Then test a replay and watch whether the backlog actually drains after the burst. Backoff can make failures less noisy without fixing the problem if incoming work keeps arriving faster than it can be processed.
Reply to this answer
Make the recovery path safe too. If a WebJob retries or a Databricks job is rerun, check whether it can send the same operation twice. Use idempotent operations or deduplication where possible, and keep track of what failed so a partial run can be recovered without blindly starting everything over.
Reply to this answer
I'd also look at how the incident was run. Having people dig through separate logs and paste clues into chat can get messy fast. Put the throttling signals, job failures, queue depth and event volume somewhere easy to see together, and record the timeline and findings afterward.
A focused incident can feel rewarding, but it shouldn't depend on skipped meals, interrupted vacations or the same people rescuing the system every time.
c.846~buttonfunnel:
Alerting on rising backlog and throttling could also give the team a chance to act before the downstream jobs fail.
Reply to this answer
Add your answer
Answers are reviewed before publication.
7c!corkwinder_624:
And check that retries share the same rate controls as new work. Otherwise the retry traffic can make the overload worse.