How Vimeo Built AI Subtitles: Solving the LLM Slot Timing Paradox in Video Translation
Why fluent LLM translation breaks video subtitle timing, and how Vimeo engineered a three-phase pipeline to keep every subtitle slot synchronized.
On this page
Video subtitles appear deceptively simple to the viewer. White text appears at the bottom of the screen, tracks spoken dialogue, and disappears cleanly when the speaker finishes.
Behind that simple experience lies a rigid, unforgiving temporal contract: the subtitle timeline is a hard grid of synchronized time slots.
When Vimeo set out to build automated AI-powered video translation across dozens of global languages, they encountered a bizarre, maddening failure mode: the more intelligent and fluent the language model was, the more catastrophically it broke video subtitles.
Translating a video through a frontier LLM produced beautiful, grammatically flawless prose.
Yet, when rendered on screen, subtitles would randomly vanish for 8 seconds during active speech, reappear out of order, or bunch up into unreadable walls of text.
Vimeo discovered the LLM Slot Timing Paradox: standard language models optimize for semantic fluency across whole paragraphs, completely destroying the rigid temporal segmentation required by video subtitles.
To solve this, Vimeo engineered a three-phase AI translation pipeline that forces language models to preserve frame-accurate time slots while delivering natural, localized translations.
+-------------------------------------------------------------------------------+
| THE SUBTITLE TIMING PARADOX |
+-------------------------------------------------------------------------------+
Source Audio Subtitle Grid (English):
Slot 1 [00:01 - 00:03]: "The weather today"
Slot 2 [00:03 - 00:05]: "is surprisingly cold"
Slot 3 [00:05 - 00:07]: "for this time of year."
Naive LLM Translation (Spanish - Optimizes for Fluency):
"El clima hoy está sorprendentemente frío para esta época del año."
* Problem: The LLM merged all three slots into ONE single sentence!
* Video Player Failure: Slot 1 displays the entire sentence.
Slots 2 and 3 remain COMPLETELY BLANK while the actor continues speaking!
1. The Subtitle Timing Contract
Traditional document translation cares only about semantic accuracy. Video subtitle translation operates under three strict physical constraints:
- 1:1 Slot Alignment: If the source language audio is split across 50 timed subtitle slots, the translated output must occupy exactly 50 corresponding slots.
- Reading Speed Ceilings (CPS): Humans read at an average rate of 15 to 20 Characters Per Second (CPS). If an Italian translation expands by 40% (the classic “linguistic expansion” phenomenon), the subtitle text will flash on screen too fast for the human eye to process.
- Line-Length Budgets: Subtitles must not exceed 2 lines of text (maximum 37 to 42 characters per line) to prevent blocking visual video content.
When you feed a raw SubRip (.srt) file into a standard frontier LLM, the model’s internal attention mechanism sees the fragmented text and instinctively acts like a helpful editor: it corrects fragmented speech, merges clauses, and outputs fewer, longer sentences.
In a document, this is brilliant; in a video player, it leaves the screen blank.
2. Vimeo’s Three-Phase Translation Architecture
Vimeo rejected naive zero-shot translation and structured their subtitle engine into three decoupled execution phases:
+-------------------------------------------------------------------------------+
| VIMEO'S THREE-PHASE SUBTITLE PIPELINE |
+-------------------------------------------------------------------------------+
[Source Video Audio Stream]
|
v
[Phase 1: Audio Transcription & Forced Alignment (Whisper + CTC Segmentation)]
- Generates frame-accurate start/end timestamps per word.
- Splits text into optimal ergonomic reading slots (CPS <= 17).
|
v
[Phase 2: Context-Aware Structural Translation (The Slot-Preserving LLM)]
- Passes full video transcript as context for semantic coherence.
- Enforces strict Slot-Preservation Prompts with unique XML delimiters.
|
v
[Phase 3: Deterministic Post-Processing & Temporal Retiming]
- Validates slot count parity: Output_Slots == Input_Slots.
- Auto-condenses overflowing lines via iterative shortening prompts.
- Flushes frame-accurate WebVTT / SRT files to CDN edge.
3. Phase 2: Enforcing the Slot-Preservation Protocol
To prevent the LLM from merging lines while still giving it the broader narrative context needed to choose accurate words, Vimeo engineered a Delimited Slot Protocol:
You are a professional video subtitle localization engine.
Translate the following English subtitles into Japanese.
CRITICAL INVARIANTS:
1. You MUST emit exactly one <slot id="N"> translation for every source slot.
2. Never merge two slots together.
3. If an English sentence spans across three slots, translate it so the Japanese
sentence fragments naturally across those exact three slots.
4. Keep character counts below 16 characters per line for Japanese.
<source_context>
Full transcript provided here so you understand pronouns, tone, and subject matter...
</source_context>
<slots_to_translate>
<slot id="1">The weather today</slot>
<slot id="2">is surprisingly cold</slot>
<slot id="3">for this time of year.</slot>
</slots_to_translate>
By presenting the full video transcript in <source_context> (solving ambiguous pronouns and tone), but restricting output generation strictly to matching <slot id="N"> tags, the model preserves 100% of the temporal slot boundaries without sacrificing semantic quality.
4. Phase 3: Handling Linguistic Expansion and CPS Overflow
Certain languages (such as German, Spanish, and French) naturally expand by 15% to 30% in character volume compared to concise English source text.
When a slot has only 1.2 seconds of video duration, an expanded German translation might require a reading speed of 28 Characters Per Second, causing viewer fatigue.
Vimeo implements an automated Iterative Condensation Gate:
def enforce_reading_speed_limits(slot, max_cps=17.0):
duration_seconds = slot.end_time - slot.start_time
char_count = len(slot.translated_text)
current_cps = char_count / duration_seconds
if current_cps > max_cps:
# Prompt model to generate an equivalent concise phrasing:
condensed_text = llm.condense(
original=slot.translated_text,
max_chars=int(duration_seconds * max_cps),
context=slot.context
)
slot.translated_text = condensed_text
return slot
If a translated phrase exceeds the reading speed ceiling, the system automatically prompts a fast distilled model to synthesize an equivalent, punchy, concise translation that fits the exact physical duration of the audio gap.
Architectural Comparison Matrix
| Architectural Vector | Naive Document Translation | Vimeo Production Subtitle Pipeline |
|---|---|---|
| Input Unit | Monolithic text document | Segmented timestamped slot grids |
| Temporal Alignment | Ignored (Zero time awareness) | 1:1 Slot Parity verified by deterministic gates |
| Context Window | Paragraph context only | Two-tier: Full video narrative context + Slot tags |
| Reading Speed (CPS) | Unbounded | Strictly capped at under 17 Characters Per Second |
| Blank Screen Failures | Common (Merged sentences) | Completely eliminated |
Summary and Key Takeaways
- Fluency conflicts with subtitle timing: Language models naturally merge fragmented speech into clean sentences, breaking video subtitle slot alignment.
- Decouple context from execution: Provide the full video transcript as background context, but enforce strict XML slot delimiters for output generation.
- Enforce 1:1 slot parity programmatically: Never trust an LLM to maintain line counts; validate slot arrays deterministically in a post-processing verification stage.
- Monitor Characters Per Second (CPS): Linguistic expansion in languages like German and Spanish will overwhelm viewers unless automated condensation gates enforce reading speed limits.
- Video translation is an ergonomics challenge: Subtitle localization is not merely translating words; it is designing comfortable, synchronized reading experiences for human eyes on moving video frames.


/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.