I'm running several 24/7 service scripts that produce a massive amount of logs. These are real-time and parsing services, which means sometimes the processes can hang, lose connections, or degrade gradually without crashing entirely. I'm looking for an effective way to centralize logs from all these services, quickly check which ones are healthy versus those that are degrading, and avoid the hassle of manually checking tons of log files. I'm currently considering two main approaches: using a logs-first setup with Grafana and Loki, or opting for a more comprehensive ELK/OpenSearch stack. All of my services are self-hosted and I'm managing everything without Kubernetes. For anyone who's tackled similar setups, what would you suggest I try first, and what trade-offs should I anticipate?
5 Answers
Have you checked out VictoriaLogs? Its agent offers some cool pre-ingestion features. You could also consider using the otel collector for enhanced data handling.
I’d recommend looking into cloud solutions if you're open to it. They often have built-in logging features that can save you a ton of hassle.
While cloud might be tempting, just keep in mind it can get pricey. Plus, if you're logging every little thing from on-prem, you might face performance issues if the cloud service is far away.
We maintained 30 days' worth of logs for a large company using an ELK stack. Be prepared for terabytes of data though; it adds up quickly!
A good strategy is to keep logs close to the sources where they're generated. You could implement a central query layer to pull logs when needed, instead of constantly sending everything elsewhere.
Splunk really is top-tier, but yeah, it comes with a hefty price tag. ELK is a solid alternative, and trying out Grafana with Loki could be a smart move for your servers. It's commonly used for monitoring and alerting, so it's a familiar territory.
I've heard great things about FluentBit too, so it might be worth looking into that along with VictoriaLogs.