I'm curious about how others handle AWS maintenance notifications and impending EC2 reboots. In our setup, we have around 100 EC2 hosts spread across three VPCs. We frequently receive emails from AWS about scheduled maintenance, direct connect changes, and various reboot notifications. Recently, I set up some automation in Gmail to catch these AWS emails and create calendar events and Slack alerts for the teams so they stay informed. However, I haven't implemented a system for reboot notifications yet. Last Saturday, we missed an important reboot email, and by Monday afternoon, our production service went down due to that oversight. I'm wondering how other systems administrators deal with these notifications. Do you automate your processes? It would be great if AWS offered more control over maintenance schedules, particularly to limit them to weekends. Any advice or strategies would be appreciated!
5 Answers
It sounds like you're relying too much on those individual instances. Ideally, your production systems should be resilient enough that the loss of one instance doesn't impact your users. If you're all in EC2, consider running systems in parallel with load balancers to manage failures more gracefully.
You might want to look into automated alerts through tools like PagerDuty, which can help manage and escalate notifications to the right on-call teams more efficiently. It’s better than scrambling to catch emails.
The key is to architect systems properly. Instances should be treated as disposable—more like cattle than pets. If you're facing regular reboots, consider using auto-scaling groups (ASGs) to handle replacements automatically when instances go down. I get emails too, but mostly I just forward them to our ticketing system. I’m looking into automating that with AWS Notifications API so we don’t miss anything in the future.
Absolutely! The AWS Notifications API can really streamline your process. It’s better to automate than depend on emails, which can easily slip through the cracks.
In our case, we have a team dedicated to managing these notifications. With over 300 EC2s, missing a few reboots isn't a problem since we have a multi-tier setup. But proactive alerts help us quickly address issues if they arise. A district email system works well for us.
As someone who used to work on the EC2 Maintenance team, I can share that AWS provides lifecycle events for these situations. You can set up Lambda functions that react to events about maintenance schedules. Check out the AWS Health Dashboard and integrate it with EventBridge for real-time notifications.
That’s gold! Custom maintenance windows for my EC2 instances would allow me to avoid those pesky weekend reboots.
But sometimes, it’s tricky for legacy systems that have been in place for years. Transitioning to cloud-native architectures isn't always feasible right away.