What’s the best way to add live captions and translation to a LiveKit stream?

You may not need to build the whole pipeline yourself. LiveKit Agents can run a server-side agent that subscribes to room audio, performs speech recognition, optionally translates the result, and publishes text back to the room through the transcription text-stream topic. Clients can listen for interim and finalized segments, using the final-status metadata to replace temporary captions cleanly. JSON-formatted chunks can also include timing information, which is useful for synchronizing captions with playback. For multilingual output, keep each language in its own topic or track so viewers can switch languages independently. The live-translated-captioning example is a useful reference for the speech-to-text, translation, and per-language flow. In practice, translate finalized or stabilized utterances rather than individual words, since complete phrases generally produce better translations.

A common approach is to have a server-side participant subscribe to the speaker’s audio track and forward it to a streaming speech-to-text service such as Deepgram or AssemblyAI. Send the returned transcript segments to the browser through a LiveKit data channel or a separate WebSocket. For translation, pass stabilized transcript chunks to a translation API. Avoid translating every tiny interim fragment; grouping words into short phrases or sentence-like chunks helps keep latency, cost, and caption rewrites under control.