If most usage is in English, an overall evaluation score could hide poor performance in other languages. How do you build test sets and decide which languages need more coverage? Do you translate the same cases, write native-language cases, or use both?
How do you evaluate multilingual LLM quality in production?
Asked by
d907+hingepin_4
Answers
Report results by language, not just as one overall score. A traffic-weighted number can be useful for seeing the aggregate, but it can look healthy while a smaller language group gets much worse answers.
Keep a separate set of cases for each supported language where you can. Cases written naturally in that language can catch idioms, phrasing and other issues that a direct translation of an English test may miss.
Reply to this answer
Don't allocate every test case strictly by traffic share. If a language has fewer users but a disproportionate number of complaints, give it extra evaluation coverage so you can investigate what is going wrong.
I'd keep the per-language results visible alongside any traffic-weighted aggregate. That way, oversampling a problem area doesn't make the overall number misleading.
Reply to this answer
Plan for language review capacity too. If nobody on the team can reliably assess a language, a test set alone won't tell you much about whether responses sound natural or actually meet the request. That constraint may affect which languages you can cover well.
Reply to this answer
Add your answer
Answers are reviewed before publication.
x418!foldmark_9:
Translated cases can still help compare how the model handles a similar task across languages. They just shouldn't be the only coverage.