I recently joined a team that uses DataDog for logs, monitoring, and alerting. I'm more familiar with Grafana, especially its alert grouping features. We have a private monitoring location inside our network that runs many synthetic HTTP tests. When the local server became unreliable, timeouts caused dozens of synthetic monitors to flap between alerting and recovery states, resulting in roughly 300 email notifications over three hours.
The synthetic tests create their own monitors, so I can't easily combine or customize them like multiple queries within a single monitor. I considered using tags for grouping, but separate monitors still generated separate alerts. Composite monitors also seem limited to a small number of monitors.
What's the best way to prevent this kind of alert storm in DataDog? Ideally, I'd like one summarized notification when a shared monitoring location fails, while also filtering out brief HTTP timeouts and flapping recoveries.
3 Answers
The main issue is probably the shared dependency rather than the individual synthetic tests. If all of the tests rely on the same private-location host, create a monitor for that host or location itself. When it becomes unavailable, use that signal to trigger a dependency-based downtime or composite condition that suppresses the monitors behind it. That prevents hundreds of downstream alerts instead of trying to group them after they have already fired.
You should also increase the evaluation window and add an appropriate recovery or no-data delay to the synthetic tests. A single short timeout should not immediately produce an alert and recovery notification if the location is known to flap.
A few techniques can reduce the noise even if native cross-monitor grouping is limited: require the failure to persist for a couple of minutes, add a recovery delay, and avoid notifying on every brief state change. You can also route the first notification to a less urgent channel and escalate only if the failure continues.
Another option is to publish an aggregate health metric for the synthetic tests or private location. The metric could represent the number or percentage of healthy tests, and one monitor could alert on that aggregate. This moves the grouping logic into the metric rather than relying on every synthetic test to notify independently.
Tags generally help with scoping actions such as downtimes and muting; they do not necessarily turn alerts from separate monitors into one grouped notification. Monitor grouping is mainly useful when a single monitor contains multiple grouped queries or alert groups, so tagging independently generated synthetic monitors will not automatically combine their notifications.
For a large set of synthetic tests, consider applying a scoped downtime to all monitors associated with the affected private location when the location-health monitor fails. That is usually more practical than building a large composite monitor.
That makes sense. The private-location host was the common failure point, so monitoring it directly and suppressing dependent tests would address the root cause better than trying to combine all the resulting alerts.