We're managing two hosted PBX server clusters that churn out around 200GB of logs each month. With 35 Linux servers involved, sifting through these logs has become a real hassle, and we're hoping to find a way to centralize our log search in one place. We're not keen on storing most of the logs since a lot of it is just noise, but we need a system that can handle significant throughput because our biggest concerns will be CPU and RAM usage.
A tricky part is that our logs aren't formatted like standard syslogs. Instead, they come in multi-line text blocks, often containing XML documents detailing log events. For instance, we have entries that look like this:
2026.04.18 12:09:49:604 EDT | Info | OCI-P | BCCT Worker #3 | some_data
...
And there are larger logs, such as entire phone directories, that we need to somehow filter out the unnecessary information from.
We're exploring applications that could receive, parse, and organize these logs for easy searching and categorization. Elastic was one option we looked at, but it turned out to be quite pricey, so we need to evaluate if we can host a solution ourselves.
Ideally, we want a local server that handles the bulk of the log parsing and forwarding, with a browser-based interface for searching. While we're open to cloud solutions, we'd appreciate any recommendations that align with our needs.
I've considered setting up an ELK stack, but I'm unsure if that's the right fit for our use case. I'm open to any and all suggestions!
6 Answers
Is there a specific vendor or ecosystem tied to your PBX? It might be worthwhile to consult them. Otherwise, it sounds like you'll need to build a custom solution. We once normalized web logs into a searchable database, and it worked well. You’d think your scale would have some direct logging options to a database like PostgreSQL.
Check out solutions like Graylog, Wazuh, or SecurityOnion. They can definitely handle what you’re looking for!
Thanks! I've heard of Graylog but wanted some real-world feedback before diving in.
For the volume you're dealing with, consider using Loki with Grafana. It’s a cost-effective way to scale and handle log aggregation without the heavy resource needs of Elasticsearch. Setting up promtail on your servers lets you search logs from one interface, which could save you a lot of hassle!
I’ve seen Loki mentioned before! I thought it was just a front-end for Broadworks. I’m glad it’s a different app.
Are there any legal requirements for keeping your logs? If not, you might want to gzip them every week; it can cut back on storage by as much as 90%. Just check how often you're actually unzipping them later, you might find you don't need to keep everything!
We don't have any legal obligations, but we do need to maintain logs for our SOC 2 Type 2 report. It's really about finding a rapid way to search through logs across multiple servers for incident responses, not storage issues.
If you could filter out the unnecessary stuff first, how much do you think you'd reduce your log size? Just curious if you've estimated the total once you strip it down.
I think if we cut out all the fluff, we might get down to about 50GB a month, which would be 25% of what we currently generate.
Have you thought about adjusting log levels? That might help reduce the volume of logs you're dealing with right now.
It’s a Cisco Broadworks PBX. We’ve talked to a third-party firm for guidance, but Cisco doesn’t offer much help directly. I’ll certainly reconsider direct logging to the database now!