Is logging enough for troubleshooting, or should we store session messages?
Asked by
CuriousCat42
I've been dealing with frustrating troubleshooting situations lately. When customers search for flights and see that they can check a bag for free, but later find out that's not the case when booking, it leads to long nights trying to uncover the issue. We often add more logs hoping to catch similar issues. One of my colleagues decided to dump all system messages into a database for troubleshooting, and while I initially thought it was excessive and costly, I've realized it's actually quite helpful. We use this data not only for fixing issues but also for analytics, such as understanding fare ranking errors and baggage rule changes. We're currently storing these messages for 30 days asynchronously to minimize latency. This experience has made me wonder, is this method too extreme?
Answers
You know, dumping logs into a database sounds a bit overkill to me. Why not just log the necessary events and then write them to files? You can store them in S3 and query them with Athena instead. Plus, it could save you the hassle of dealing with database overhead.
Reply to this answer
You might want to consider using JFR (JDK Flight Recorder). It’s built right into the JDK and captures a ton of useful events without overwhelming you. You can set thresholds so it only records relevant data, which helps keep the noise down. Plus, it works asynchronously too, just like you mentioned! Here’s a good intro to using it with JDK Mission Control: [JFR Intro](https://www.youtube.com/watch?v=7-RKyp05m8M).
Reply to this answer
From my experience, every company I've worked for has some sort of log storage system, whether it’s ELK stack, Datadog, or a custom setup. It’s unusual not to push logs somewhere. But I’m curious, what exactly differentiates your session messages from application logs? Why not just use the same system for both?
Reply to this answer
Certain messages definitely hold business value, and storing them can save a lot of headaches later. If you ever face legal issues, having reliable logs can make all the difference. If I were starting a new project, I’d integrate structured logging from the get-go. It’s essential for tracking what's going on, especially in distributed systems. You can connect logs with a request ID to see the bigger picture.
Reply to this answer
It sounds like you’re on the right track with the storage approach, especially if you’re using the data for both troubleshooting and analytics. Just be mindful of what you're logging. Logging should serve a purpose — focus on what's beneficial for debugging. Have you thought about tools like Grafana or Datadog? They can help manage logs more efficiently without flooding your database.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions