How Can I Better Manage Log Files to Prevent Disk Overfill?
Asked by
TechWizard42
Recently, my app server ran out of disk space in /var due to log files, causing some services to break down. I thought logrotate would manage this since we already had it set up. However, I found that a few services were logging to unexpected paths, and one application had debug logging enabled, which overwhelmed the log rotation process. I've cleaned up the situation and added rotation for the missing paths, but this issue keeps arising in different forms. I'm questioning whether relying solely on logrotate is sustainable long-term. Do you implement logging controls at the application level, such as limits or using just stdout, or do you enforce strict logging paths and rotation management from a central point?
Answers
I have all logs routed to a central SIEM server. Local machines, regardless of whether they are running Linux or Windows, can quickly fill up, so you need monitoring for disk health and proactive alerts to catch anything not following your SIEM's export rules.
Reply to this answer
Logrotate can definitely be helpful, but it sounds like you're facing two main issues: one related to configuration management and another concerning observability. If your apps don't handle their own log rotation, then logrotate is a solid solution. It's crucial to have proper monitoring and alerting, so issues like disk fills get flagged before they cause problems. Setting up a tool like Zabbix can help notify you through Slack or similar channels when major events occur.
Reply to this answer
I recommend using a monitoring system to catch these situations before they escalate. You want to be alerted early to avoid disk issues.
Reply to this answer
If you can enforce specific log locations and rotation, it’s best to do that. However, be aware that if your apps are writing logs unexpectedly, you'll need to pinpoint those specifics. Understanding how each application interacts is crucial. Fix one app, and that should resolve that part of the problem. Also, ensure apps run under appropriate user permissions to prevent unexpected log writes.
Reply to this answer
Add your answer
Answers are reviewed before publication.
QuickFixAdmin: Exactly! We don't have proactive alerts for disk usage either, so we end up reacting instead of preventing these issues.