Hey everyone! I'm looking for effective strategies to reduce noise in my observability data, particularly in logs. We're currently overwhelmed with low-signal logs coming in, especially from info and debug levels, and it's making it difficult to identify real issues while also increasing our storage costs. We've tried some basic filtering techniques and have asked developers to reduce logging, but the noise just keeps coming back. I'm curious if anyone has found reliable methods or tools to help manage this issue. I'd love to hear your experiences and any success stories, plus any horror stories you might have as well! Thanks!
1 Answer
Hey there! This noise issue is definitely a challenge, but I've got some tips that might help. First off, try logging at the edges of your systems rather than deep in the core. This means logging at the route or controller level instead of everywhere in your DB helpers, which can help reduce log volume and improve clarity.
Also, consider switching to structured logging. Using key/value pairs instead of plain string logs makes it a lot easier to filter out unnecessary data. If you’re using OpenTelemetry, think about setting up processors to drop high-volume logs based on regex or attributes. This can really streamline your logs! Hope this helps! By the way, I even wrote a blog on this—feel free to check it out!
Thanks for sharing your insights! But do you have any suggestions to make the process less tedious?