Two eu-west-1 instances failed network checks at the same time
Asked by
MellowCedar42
Two instances in the Dublin region went down together around 2:40 PM. Both reported failed network-reachability health checks, but there was no corresponding incident on the service health dashboard. They were in the same availability zone, so I wondered whether an underlying host, rack, or network issue could have caused both failures. Restarting the instances resolved the problem, so this is mostly a curiosity: has anyone seen a similar localized failure, and would this type of event normally appear on the health status page?
Answers
Since the instances were in the same availability zone, check their placement and host relationship if you have access to that information. They may have shared a physical host or nearby infrastructure, which would explain why they failed at the same time. A physical host failure is another reasonable possibility.
Reply to this answer
A rack-level failure is a plausible explanation. If the two instances happened to be placed together, a problem affecting that rack or nearby host infrastructure could take them down without affecting the entire availability zone. Smaller infrastructure incidents do not always generate a public health event.
Reply to this answer
It could also have been a brief networking problem, a small power interruption, or an infrastructure component being disconnected or serviced. Scheduled maintenance would usually come with advance notification, but not every short-lived physical or networking fault is reported individually.
Reply to this answer
There does not seem to have been a broader regional problem; another operator with a large fleet reported no impact. If you need a definitive cause, opening a support case is probably the only way to get provider-side host or infrastructure details, although a short outage that recovered after a restart may not be worth escalating.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowCedar42: That was my thinking too. A short note on the health dashboard would still be useful, even just to confirm the issue was on the provider side rather than something we did.