Hey everyone! We're developing a backend and app for a large retailer with thousands of stores, which means we have about 10,000 servers scattered worldwide. We're currently facing a challenge with our logging system. On one hand, we need to centralize logs for monitoring, but we also want to keep costs manageable—around several million euros per year. On the other hand, our team often receives feedback that our logs aren't detailed enough when bugs arise, but adding more logs might push us over budget.
I'm considering implementing a decentralized logging system where each server maintains its own local log storage, but also sends key logs to Elastic for central monitoring. Importantly, we don't want to connect to each server individually, especially since they are all running Windows. Does anyone know of a system that allows decentralized log storage along with a centralized management approach?
1 Answer
If local log storage isn't a strict requirement, consider sending your logs to cloud or blob storage. Keeping logs distributed might complicate your setup, especially with security and maintenance concerns, given you have 10,000 targets. Cloud options can provide more flexibility for querying based on the provider you choose.
That’s a solid suggestion! I hadn’t thought about blob storage. I came across Loki by Grafana, which seems like a viable low-cost alternative. Our client insists on local logs due to their specific reliability needs, but it’s definitely worth exploring!