I'm currently evaluating our on-call process because I've noticed we're wasting a lot of time on manual triage. The main issue arises when an incident occurs after hours. Typically, someone gets called up to check if a Slack alert corresponds with an email from a high-priority client, identify the service owner, and then decide whether to escalate it or wait until morning. While I believe the decision-making process (Severity + Client Tier + Service Impact) is quite straightforward, we're still relying on people for routing. Has anyone been able to automate the decision-making process between incoming channels (like Email, Slack, and PagerDuty) and the actual response, like creating a Jira ticket or escalations? Or is the risk of an automated system misclassifying a critical issue simply too high? Am I overlooking a useful tool, or are others facing similar challenges?
5 Answers
The first thing you should do is reduce the number of services you're monitoring for alerts. If you can consolidate down to just one, automation will become much more manageable. Keeping everything centralized simplifies the process.
You're looking at this from the wrong angle. If a client messages outside of regular hours, the response should be straightforward: get them on the phone, loop in an engineer, and start the billing clock. Issues that can wait until morning shouldn't trigger alerts. Streamlining your processes will save unnecessary stress.
I've seen that most teams just accept the manual process until they hit a breaking point. Usually, someone in engineering is bribed to set up a webhook, but it often ends up breaking in production after a few months due to lack of documentation. If you really want to do this effectively, consider writing your severity rules as code (like in YAML) rather than relying on a rigid vendor interface that will be hard to maintain. Test this against past incidents and include a human override feature for misrouted alerts—this will reduce misclassification significantly.
The most effective solution I've encountered is using a 24/7 outsourced phone line that triages calls. If the severity is deemed high enough, they can page the appropriate on-call person. This way, the triage is done efficiently and only the critical issues escalate to the on-call staff.
In my setting, we have a helpdesk operating most hours. When they're off, the on-call person simply gets a call to assist with incidents. This has worked well for years, so I see little benefit in complicating matters. If you truly require 24/7 support, it might be worth investing in a dedicated team. Otherwise, you might just need to ensure someone is consistently monitoring your support system.

Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures