Recently, my app server ran out of disk space in /var due to log files, causing some services to break down. I thought logrotate would manage this since we already had it set up. However, I found that a few services were logging to unexpected paths, and one application had debug logging enabled, which overwhelmed the log rotation process. I've cleaned up the situation and added rotation for the missing paths, but this issue keeps arising in different forms. I'm questioning whether relying solely on logrotate is sustainable long-term. Do you implement logging controls at the application level, such as limits or using just stdout, or do you enforce strict logging paths and rotation management from a central point?
4 Answers
If you can enforce specific log locations and rotation, it’s best to do that. However, be aware that if your apps are writing logs unexpectedly, you'll need to pinpoint those specifics. Understanding how each application interacts is crucial. Fix one app, and that should resolve that part of the problem. Also, ensure apps run under appropriate user permissions to prevent unexpected log writes.
I recommend using a monitoring system to catch these situations before they escalate. You want to be alerted early to avoid disk issues.
Logrotate can definitely be helpful, but it sounds like you're facing two main issues: one related to configuration management and another concerning observability. If your apps don't handle their own log rotation, then logrotate is a solid solution. It's crucial to have proper monitoring and alerting, so issues like disk fills get flagged before they cause problems. Setting up a tool like Zabbix can help notify you through Slack or similar channels when major events occur.
I have all logs routed to a central SIEM server. Local machines, regardless of whether they are running Linux or Windows, can quickly fill up, so you need monitoring for disk health and proactive alerts to catch anything not following your SIEM's export rules.
Exactly! We don't have proactive alerts for disk usage either, so we end up reacting instead of preventing these issues.