Hey everyone! I'm working as an R&D AI Engineer, and I'm trying to establish a CI/CD pipeline to streamline our development process and save time for my team. Recently, I set up a pipeline that runs evaluations whenever there's a change in the evaluation dataset, but I'm running into some challenges and uncertainties about best practices.
Specifically, I'm looking for advice on two things:
1. How can I effectively track the history of evaluation results alongside module versions (which could include prompt versions and LLM configurations)?
2. What tools are recommended for exporting results to a dashboard?
I'm sure there might be other important aspects I haven't considered yet, so I'd love to hear how your teams handle this. Thanks a bunch!
4 Answers
It sounds like you're navigating both R&D and practical development, which can be a bit tricky. Just remember to tag everything obsessesively! It'll help you track changes better.
Have you thought about using git tags or branches for tracking your history? That could help with version control. For dashboarding, popular tools like Grafana or Kibana are really solid choices—they make visualization pretty straightforward!
Just a heads-up, the field is moving towards what they're calling 'Engineering Intelligence' these days. We're gathering insights and scores from our IDP Port with more observability features. It could be worth looking into as part of your strategy!
We made something similar at our project. For tracking evaluations, we used MLflow because it takes care of the versioning complexities really well—definitely useful for managing your nested configurations. We also dumped our data into a PostgreSQL table with jsonb for those nested configs and then used Grafana on top of that. Plus, a tip: make sure to save the whole config snapshot with each evaluation run. It'll save you a ton of headaches down the line when trying to troubleshoot drops in performance!
This is awesome! Thanks for the suggestion, I'll definitely try it out!