What’s the best way to add real-time translated captions to a LiveKit stream?

You can subscribe to a participant’s audio track on the server and stream it to a speech-to-text provider such as Deepgram or AssemblyAI over a WebSocket. Send the returned transcript segments to the browser through LiveKit data messages or a separate WebSocket. For translation, pass the stabilized transcript into a translation API. Avoid translating every tiny partial update—buffering a few words or waiting for a short phrase helps keep latency and API usage under control.

LiveKit’s Agents framework can handle much of this pipeline without building every component yourself. A server-side agent subscribes to room audio, runs speech-to-text and an optional translation step, then publishes the resulting text through the transcription text-stream topic. Client SDKs can distinguish interim and final segments using the transcription-final metadata. Enabling JSON-formatted output can also provide timing information for better playback synchronization. For multilingual captions, the official live-translated-captioning example demonstrates the speech-to-text, translation, and per-language caption flow. In practice, translate complete sentences or natural utterance boundaries rather than individual word fragments, since that generally improves both quality and stability.

Treat captions as timestamped events instead of ordinary chat messages. A server-side participant can subscribe to the room audio, send it to speech-to-text, and publish partial and final segments containing a speaker identifier and sequence number. The frontend should update interim text in place while preserving finalized captions. Translation is usually better on finalized or stabilized utterances, and the original and translated captions should be kept as separate channels so viewers can switch languages.