I'm evaluating a new monitoring system for servers and network equipment, and it's making me reconsider sending every alert straight to email. How do you route alerts in your environment? Do you separate destinations by severity or type, and what channels actually get people's attention?
4 Answers
The key is tuning the rules before choosing the destination. Our monitoring system sends selected warnings and errors into the ticketing platform, while critical events trigger push notifications or phone calls. A dashboard is great for reviewing trends over coffee, but an alert should only exist when someone needs to take action promptly.
There isn't one destination that works for everything. Critical alerts should page someone through a system that can call, text, or push-notify them. Medium-severity issues can create tickets or go to a team channel, while informational events belong in email, a dashboard, or an event history. Anything that doesn't require a defined action probably shouldn't be an alert at all.
That severity-based approach makes sense. I'm trying to avoid creating one noisy stream that people eventually ignore.
Email is useful as a record, but it scales badly for real-time alerting. We use a shared alert address or mailbox for lower-priority notifications and connect useful events to the service-management system so they become tickets. Make sure the system deduplicates alerts so one outage doesn't create hundreds of tickets.
For important incidents, we send notifications to a collaboration channel where the team can discuss and acknowledge them. The truly urgent ones also page or push directly to on-call staff. Informational alerts go to a dashboard or email folder, where they can be reviewed without interrupting everyone.
Exactly. If an event isn't actionable, it belongs in history or reporting rather than an interruptive alert channel.