I'm trying to implement a reliable alerting system for my critical EC2 infrastructure. The issue is that sometimes it might take me up to 30 minutes to realize something has gone down. Ideally, I want to be notified within five minutes if any critical EC2 instance shuts down or becomes inoperable. I initially thought using a CloudWatch alarm that triggers when CPU Utilization is at 0% for five minutes would work, but when I tested it with a shut-down instance, I didn't receive any alerts from SNS. What's the best way to set this up? Also, I noticed that the alarm state showed 'Insufficient data,' which suggests my current approach isn't working since it relies on the instance being operational.
7 Answers
Have you thought about configuring your CloudWatch alarm to trigger an alert for missing data? No CPU Utilization data means the instance is likely down, so that could work for your needs too!
If you're running a web server or any service, I'd recommend using a tool like Better Uptime to ping and alert you. It's super easy and effective. After that, you might want to set up autoscaling with health checks to automatically recover the instance if it goes down. It takes a bit more setup but it's worth it in the long run!
If you want to specifically monitor when an instance stops, consider adding an EventBridge rule to react to state-change events and send an SNS message when that happens. It will give you email alerts if the instance is stopped or in trouble. Also, consider adding status checks to your instances for extra monitoring.
We use a service called Better Stack, and before that, we relied on Pingdom. If you can set up a public endpoint, then you have even more options like Uptime Robot for reliability and alerts.
Check out EC2 autoscaling lifecycle hooks to see if you can integrate those for better alerting and management!
Definitely go with EventBridge. I've seen people suggest CloudTrail for monitoring, but its reliability can be iffy since it only tracks API calls. If the instance is shut down directly from the OS, you won't get notified with CloudTrail.
You might want to look into using EventBridge to send notifications through SNS. You’ll need to figure out which EC2 instances will trigger the alerts, but it’s a pretty straightforward solution. Just set the rules right!
Totally agree! Automating the recovery process is key. Instead of just waiting for issues, proactively fixing them can save a lot of hassle.