I have a background process running on my EC2 instance that dies every day at exactly 11:29 AM CDT, but I can't figure out why. I've looked through everything imaginable, including cron jobs, systemd timers, CloudWatch alarms, and more, and I found nothing that could cause this. The VM has 32 GB of RAM, and my process doesn't use more than 2 MB of memory, so memory issues aren't the cause. There's no crash or error logs indicating what went wrong. It mainly just sits there idle, only handling pings from a remote server. Has anyone encountered something like this before?
4 Answers
I'd recommend checking the syslog and dmesg around the time the process dies. There could be something recorded there that gives you a clue. Sometimes, other system activities can interfere in unexpected ways.
Could your organization have some sort of compliance or monitoring tool that's set to kill processes that aren't supposed to be running? It's worth investigating if there are any automated checks occurring around that time.
Interesting thought! I'll need to look into SSM jobs and see if anything is causing this.
It sounds like this could be some kind of timeout issue. If your process has a set limit and is running for the same duration every day, it might be getting killed due to that limit, even if it seems idle. Try increasing the logging to see if you can catch any hints during those times. Running it in a different environment could also help narrow down if the issue is with the process itself or the EC2 setup.
That makes sense! I’ll add some detailed logs to monitor the exact behavior at that time. Thanks for the tip!
You mentioned that RAM usage never exceeds 20%. That’s good, but it’s also worth confirming that there aren’t any memory leaks elsewhere that could cause unexpected behavior. Sometimes they lead to odd process deaths.
I'm pretty sure there aren't any resource leaks, but I'll monitor the system closely.
I checked the logs and didn't find anything relevant, just some routine messages. But I’ll keep digging through the logs!