I've been having an issue with my Ubuntu EC2 instance (2 GB) where it suddenly loses all network connectivity every two days around 05:30 UTC. Here's what I've observed: the systemd-networkd logs show "ens5: Could not set route: Connection timed out," which leads to my website going down, and I can't SSH into it or access the AWS web console. I usually have to manually reboot to restore connectivity, but even then the network initially shows some link flapping.
Looking at the logs, there are no hardware or driver errors detected (the ENA adapter seems fine), but I did notice that the AWS SSM agent was failing with 400 errors before the network issues occurred and the Snapd service timed out, likely since there's no network connection at that point.
Given this, I have a few questions: 1) Is this a common networking issue with AWS, or is it something I should take seriously? 2) How can I set up my system to automatically recover from these kinds of network failures? 3) Is there a way to prevent a single network interface failure from bringing down the whole server? My environment is set up with Ubuntu 22.04, Node.js, PM2, and Nginx, along with Puppeteer and Chromium-browser.
3 Answers
From what you’ve described, it doesn't sound like a standard AWS issue. It seems more like a memory leak in your application causing the Out Of Memory (OOM) killer to kick in every 48 hours. You might want to check your logs for any indications of this. A good testing approach would be to set up a cron job that automatically reboots your server at midnight each day. If that helps, then you likely have a resource leak.
Also, if you think your instance may have hardware issues, stopping and starting the instance (instead of just rebooting) can sometimes move it to a new hypervisor, which might resolve underlying issues.
Are you using a burstable instance? If so, that's worth considering. Your specs indicate it might be burstable, which could affect performance during high load times.
Were you able to SSH into your instance during those downtimes? I’ve had similar issues with Lightsail when running WordPress, but there were some different troubleshooting steps I took that helped resolve it. For instance, make sure you check the health of the instance during the downtime.
I’ll try that midnight reboot to see if it makes a difference!