I've recently integrated AI features into my app and noticed a massive increase in logging data—going from about 100MB per month in normal website logs to around 3GB in LLM conversation logs, and it's still climbing. My existing logging system is struggling with this surge, leading to issues like query timeouts and rising costs. I'm looking for ideas on how to re-architect my LLM logging setup to better accommodate a growing user base and the increasing token usage that comes with advanced models, tool usage, and multi-agent systems. How do you handle your LLM logs? Do you store them alongside application performance monitoring (APM) logs, use a dedicated LLM logging service, or build a custom solution using open-source tools?
2 Answers
We use ELK running on EKS to manage the logging. It's set up with three ARM pods and 2TB volumes, which can handle over 100GB per day for application, APM, and custom logs without any issues. We keep 14 days of logs indexed and up to 10 years archived, with some encrypted in Glacier. Logs are written in JSON, and while only some parts are indexed, the rest is stored as-is in ELK.
One approach I've seen is sending logs to Loki, with the backing storage on S3 and transitioning to Glacier after 90 days. It's a reliable and budget-friendly setup!
I'm also using this method; it works quite well for our needs.
How are you sending the logs to Loki? Are you using OpenTelemetry? Also, how is it handling the difference between LLM requests and responses? How do you keep track of conversation history for threads?
Just so you know, Glacier isn't officially supported. If you're putting old logs there, how do you manage restoring specific chunks when needed?
Do you log the LLM requests separately from the responses? Do you keep the entire conversation history, or just the latest user input with the response? Also, how do you query this data? Can you perform SQL queries or text searches? It would be beneficial for analyzing conversations, like finding instances where users ended up frustrated.