Hey everyone! We're in the process of developing a backend and app for a large retail chain consisting of around 10,000 store servers worldwide. We're facing a challenge with logging due to two conflicting needs: we want all logs to be centralized for monitoring (which is costly, as we're using Elastic and looking at several million euros annually) while also ensuring that the logs are detailed enough for debugging when issues arise. Adding more logs might violate our budget constraints. One idea we're considering is creating decentralized log stores, where each server maintains its own logs locally but sends the most critical logs to Elastic for central monitoring. I'm looking for recommendations on systems that can facilitate decentralized log storage while allowing centralized management functionality. We prefer not to connect to each server individually, especially since they're all running Windows. Any insights or experiences you can share?
1 Answer
If you don't necessarily need to keep logs on-site, consider shipping them to cloud or blob storage. Keeping logs decentralized seems unnecessary unless you have someone on-site who needs them. Once they're in blob storage, you can take advantage of various querying options depending on the provider.
That’s a valid point; I hadn’t thought of blob storage before! I found Grafana's Loki, which could be a good alternative as a cost-effective solution. Our client prefers having logs on-site due to their requirements for reliability and network security is already in place.