I manage two hosted PBX server clusters that generate a staggering amount of logs — around 200GB per month. We're seeking a way to consolidate and search these logs from approximately 35 Linux servers since manually sifting through them is quite a hassle. We don't intend to store most of these logs, as a lot of it is just background noise that we can discard. The solution we choose needs to handle significant throughput, so the focus will be on CPU and RAM capabilities.
A key challenge is that our logs aren't standard syslog formatted; they mostly consist of multiline text, often containing XML documents with detailed logs. We want a system that can effectively receive these logs, extract relevant content, and enable easy searching or categorization. As a couple of examples of what we deal with:
1. "2026.04.18 12:09:49:604 EDT | Info | OCI-P | BCCT Worker #3 ..."
2. "2026.04.18 12:09:49:998 EDT | FieldDebug | OCI-P | BCCT Worker #3 ... User: Call Reporting ..."
We're considering options like setting up our own ELK stack to handle cleaning up and forwarding these logs, but we're unsure if ELK is the right fit for our use case. We're open to cloud-based solutions if they match our needs. If Elastic is the best choice despite budget concerns, we're ready to hear about that too! All suggestions are welcome.
5 Answers
Given your volume, Loki with Grafana might be your best bet for an affordable and scalable solution. It lets you aggregate logs without the resource demands of Elasticsearch. Set up Promtail on each server to send logs to a central Loki instance, and you can easily search across all servers in one spot.
Before sending your logs to a central location, could you estimate how much log data you'd have if you filtered out all the unnecessary stuff? It might help you determine what to capture before forwarding.
I’d say if we stripped everything non-essential, we could probably get it down to around 25% of the original size, about 50GB/month.
Have you checked if there's any legal requirement to keep your data? If not, you might just want to gzip the logs weekly to save space. It can reduce the size significantly, like up to 90%. Just track how often you actually need to access them after compression.
We don’t have a strict legal requirement, but we do maintain a SOC 2 Type 2 report, which requires us to keep logs for incident responses. We're not worried about storage since the PBX has an auto-archive function, but we really need to make these logs easier to search and correlate across multiple servers.
Is there a vendor for your PBX system? It might be worth talking to them first to see if they have any built-in solutions. Alternatively, you might need to develop a log pipeline that normalizes and compacts your log data into something searchable in a database.
We’re working with a Cisco Broadworks PBX, and unfortunately, they didn’t provide useful recommendations. I’ll explore the idea of logging straight to a relational database based on your suggestion!
Have you thought about using Graylog, Wazuh, or SecurityOnion? They all work well for log management and should fit your needs. Let me know what you think about those after checking them out!
I appreciate the suggestions! I’ll definitely look into those options. I tend to trust user recommendations over AI suggestions!
I’ve seen Loki referenced before but mistook it for something else. Thanks for the heads-up! I’ll definitely give it another look now.