I'm looking for a way to receive alerts whenever a deployment fails because of a lack of compute capacity in a specific region. It's important that this alerting system operates independently of any particular service. Just to clarify, I'm not interested in quota-related failures; I'm specifically focused on scenarios where a deployment fails even though there is quota available.
3 Answers
This is a tricky situation because not all platforms provide a straightforward 'capacity failure' signal. Typically, these issues manifest as vague deployment failure events. A common approach is to link into deployment logs and filter for specific error messages that indicate capacity issues, then set up alerts based on those patterns. To keep it service-agnostic, centralizing your logs and alert rules in a tool like Datadog or CloudWatch is likely your best bet.
We're struggling with this too and face these deployment issues weekly across various resource types. It's getting really frustrating! What region are you deploying to?
If you're using Infrastructure as Code (IaC) like a DevOps pipeline for deployments, you can extract failure data from the deployment step. This allows you to set up actions like emails or webhook triggers when a failure occurs. However, with multiple teams handling their own pipelines, tracking these failures becomes complex.
We manage hundreds of subscriptions across different DevOps teams, and it's a challenge for us as a centralized team to keep track of everyone’s Azure DevOps pipelines.