I'm curious if anyone utilizes Azure's storage solutions for keeping prompts and responses from the OpenAI services in a production environment. How do you handle this? Do you use Azure Monitor for tracking, or do you prefer storing data in Cosmos DB or Blob Storage? Or maybe you don't store anything at all—if that's the case, I'd love to hear your reasoning!
5 Answers
If you’re using the Langchain SDK in Python, you can enable Langsmith and you'll have a full observability dashboard in no time!
I prefer Blob Storage myself. I route all my requests through APIM to Azure OpenAI, which logs everything to an Event Hub with capture enabled for Blob.
I recommend using a dedicated LLM Ops tool. I've been using Langfuse, and it works like a charm.
Oh nice! But where exactly do you keep your traces?
I’ve experimented with logging via APIM too, and it does the job quite well!
I store everything in Cosmos DB. It's great for testing, validation, and even for legal purposes sometimes. I find it really useful for reporting and training my models too!
That's interesting! Do you have a specific library or method for managing that data? Or is it all custom-built?
What if I'm not using Langsmith? Any other options?