I suggested that we start using structured logging at work due to our complex microservices platform. However, I'm facing a lot of pushback from the team, and the ROI isn't crystal clear to them. Right now, troubleshooting issues can take us an entire day. What have your experiences with structured logging been like? Is it truly a beneficial investment?
6 Answers
I can't imagine working without structured logs anymore, especially in a microservices environment. They allow you to index log values and easily search across multiple services without getting lost in string lookups. For instance, if you include a user_id field in your logs, you can quickly find all logs related to a particular user across all services. We use Loki for log management and Grafana for visualization-it's a game changer!
When I say 'structured logs', I mean sending log entries in a format like JSON. Fields such as trace_id and session_id help correlate entries between services instead of piecing them together manually from each service.
Exactly! Having those dedicated fields streamlines the whole debugging process.
In my view, structured logging is totally worth it. Just don’t try to create your own formats; stick to established structures like ECS or OTEL.
Honestly, how tough is it to start logging in a structured way? The initial investment isn't much at all, so I don't get why there's hesitation from developers to adopt what seems like such a best practice.
Right? Currently, we log practically everything, which just creates noise. My boss thinks structured logging is a waste of time, but I pointed out it would actually help us clean up and manage what we log better. Some nights, we are debugging issues for hours just sifting through logs!
Implementing structured logs can significantly enhance visibility across the services. Fields like trace_id can be a huge help for debugging. While there’s an upfront cost and some effort involved, modern tools make this process much simpler. I’d recommend centralizing your logs first and then gradually adding structured data. It could really show its worth as your platform scales, and you might want to suggest pilot projects to demonstrate the value of structured logging to your team.
It's wild we're still debating this after 15+ years! What's there really to argue about?
Especially when the boss argues it’s not important while we're trying to juggle a bunch of new features that are in the pipeline...
We propagate a requestid for tracking too. If a user encounters an error, just give us that requestid and we can pull up all timings, subrequests, errors, and stack traces.