Building a Production AI Translation and Lip-Sync System at Scale

meta · Media & Entertainment · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production multimodal translation and lip-sync for social video

Meta needed to translate Reels into other languages, preserve speaker voice/tone, generate lip-synced video, and deliver the right language version at platform scale without creating unsafe or misleading AI media.

Constraints:

• the system had to process creator uploads through distributed media storage and asynchronous workers
• audio translation had to preserve prosody, emotion, tone, and timing across languages with different verbosity
• the pipeline had to coordinate more than ten models across audio, speech, safety, and video processing
• production content includes noisy audio, varied edits, different skin tones, and different facial characteristics rather than clean short lab samples
• translated media needed transparent AI provenance, watermarking, and controls for hallucinated or toxic translations
• delivery had to stay responsive across locale, language preference, and region-specific playback conditions

Architecture

Meta built AI translations as an asynchronous multimodal media pipeline. A Reel is uploaded to Oil distributed storage, a translation request is queued, AI-enabled workers fetch the media, run audio translation through eligibility checks, sentence splitting, Seamless translation, time stretching, background-noise restoration, safety filters, and then feed translated audio plus the original video into lip-sync preprocessing, alignment, per-Reel/language tuning, inference, muxing, watermarking, provenance metadata, and playback delivery keyed by user language settings.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  creator-upload[Creator upload] --> oil-distributed-storage[Oil distributed storage]
  oil-distributed-storage --> translation-queue[Translation queue]
  translation-queue --> ai-enabled-worker[AI-enabled worker]
  ai-enabled-worker --> audio-translation-pipeline[Audio translation pipeline]
  audio-translation-pipeline --> seamless-model[Seamless model]
  audio-translation-pipeline --> safety-and-integrity-layer[Safety and integrity layer]
  ai-enabled-worker --> lip-sync-pipeline[Lip-sync pipeline]
  lip-sync-pipeline --> safety-and-integrity-layer
  lip-sync-pipeline --> oil-distributed-storage
  oil-distributed-storage --> playback-stack[Playback stack]
  playback-stack --> consumer-device[Consumer device]

Components

Data flow

A creator uploads a Reel to Oil. Meta enqueues a translation order, an AI-enabled worker downloads the media, translates audio, creates the translated and lip-synced asset, and writes it back to Oil. On playback, the delivery stack uses the viewer's device language and locale signals to choose the original or translated asset and prefetch it for a responsive experience.

Why this architecture

Meta used an asynchronous storage-and-worker pipeline because translation and lip sync are heavy media transformations, not synchronous request/response features. Seamless provided the core translation model, while auxiliary eligibility, segmentation, alignment, safety, and delivery systems were needed because production Reels differ from clean model-training examples and because language-specific assets must be delivered responsively and transparently.

Models used: Seamless · language identification model · audio classifier models · automatic speech recognition · voice activity detection model · lip-sync model · Minto toxicity detector · auxiliary audio/video models

Scale

• The audio pipeline uses more than ten models.
  “detail audio pipeline comprises of multiple steps and uses 10 plus models for different processing steps as a first step we decode the audio to get PCM signals out next we done pre-processing stage which has multiple steps like eligibility sentence”transcript, stated
• Seamless supported six languages at the time of the talk.
  “meta released it last year it currently supports six languages it maintains procity emotions and tone during translation to ensure the translated voice doesn't seem robotic and it matches with the source voice with this model at the core we have built audio translation pipeline let's look at the”transcript, stated
• The production target is billions of videos rather than short clean samples.
  “whereas in production we process billions of videos with content length with different background noises edits skin tones and faces”transcript, stated
• The alpha rollout was limited to a very small number of creators.
  “thank you hello again so AI translations is just in the alpha stage meaning that only a very few creators have access to it but overall the feedback that we are receiving has been quite positive and we are very happy about that so you can see some examples in the”transcript, stated
• Meta reported about 90% translation and lip-sync eligibility for submitted media.
  “increase about eligibility so 90% of the media that was sent to translate and lip sync we were able to do it so we detected that the proper voice and we were able to translate it in Li syncing and what next so what are the next steps so we want to improve our system”transcript, stated

Alternatives considered (roads not taken)

Split audio arbitrarily into small chunks

Rejected because: Cutting mid-sentence can harm translation quality, so Meta used ASR punctuation and VAD pauses to split around sentence boundaries.

Leave translated audio at its natural translated length

Rejected because: Different language lengths would desynchronize audio and video, so Meta used time stretching to match source duration without sounding rushed or slow.

Rely mainly on reference-based automatic quality metrics

Rejected because: The automatic metrics correlated poorly with human perception, forcing heavier subjective eval and statistical bias removal.

Move decoded uncompressed frames through ordinary production networks

Rejected because: Shared network and memory constraints made uncompressed frame movement risky, so Meta designed streaming interfaces.

Patterns

batch-offline-processing (workflow-orchestration)

Translation requests are queued after media upload, then AI-enabled workers fetch media from Oil, transform it, and write generated language variants back to storage.

guardrails-io-filtering (safety)

The pipeline adds eligibility checks, hallucination/toxicity mitigation, watermarking, provenance metadata, deletion, and user feedback around generated translations.

eval-driven-development (evaluation)

Model iteration depends on subjective human ratings, statistical bias/variance removal, and expert analysis because reference metrics did not match human perception.

latency-optimization (delivery)

The playback stack uses language and regional signals to prefetch the right translated asset and keep viewing responsive across locations and language preferences.

Tradeoffs

Use sentence-boundary segmentation before translation

Chose ASR punctuation plus VAD-based sentence splitting over arbitrary short audio chunks.

Rationale: Seamless performs better on smaller segments, but cutting mid-sentence can degrade translation quality.

Stretch translated audio to match source timing

Chose time stretching with naturalness constraints over letting translation length determine audio duration.

Rationale: Matching source duration preserves audio/video sync, but the adjustment must avoid making speech sound rushed or slow.

Prioritize semantic translation at scale while adding lip sync

Chose AI translation plus lip-sync generation over human-crafted dubbing that optimizes lip movement at the expense of semantics.

Rationale: Meta needed scalable translation within minutes for many content types, while still avoiding jarring audio/video mismatch.

Use subjective human evaluation despite scaling cost

Chose human ratings with statistical debiasing and expert analysis over reference-based metrics as the main quality signal.

Rationale: Automatic metrics poorly tracked human perception, but human ratings are subjective, slow, and hard to scale.

Add explicit AI provenance to generated media

Chose visual watermarking and provenance metadata over unmarked translated/lip-synced media.

Rationale: Generated media needs transparency and manipulation protection, especially when it can alter voice and lip movement.

Gotchas (what bit them)

Translated speech length can break audio/video synchronization.

Root cause: Languages encode the same meaning with different word and syllable counts.

Resolution: Meta added time stretching so translated audio matches the source duration without sounding unnaturally rushed or slow.

A model trained on clean speech did not handle noisy production videos by itself.

Root cause: Real Reels contain background noise, edits, and diverse recording conditions unlike the clean training data.

Resolution: Meta extracted background noise during preprocessing and added it back during post-processing using in-house algorithms and models.

Automatic translation metrics did not reliably reflect perceived quality.

Root cause: Translation quality spans accuracy, expressivity, voice similarity, and lip/audio sync, which reference metrics correlated with poorly.

Resolution: Meta invested in subjective evals, human ratings, statistical debiasing, and subject matter expert analysis for model iteration.

Lab-model assumptions broke under production media scale.

Root cause: The original model targeted short clean voice samples, while production had billions of videos with varied length, noise, edits, skin tones, and faces.

Resolution: Meta complemented the main model with auxiliary models that were easier to tune for production variation.

Passing uncompressed frames between models can overload memory and shared networks.

Root cause: Decoded uncompressed video frames move through multiple model steps in a production network rather than an isolated lab.

Resolution: Meta redesigned interfaces and workflows, including streaming interfaces, to avoid out-of-memory and network saturation.

Evaluation

Meta evaluated the system with correlation studies, reference/reference-based metrics, subjective evals, human ratings, statistical bias/variance correction, expert analysis, eligibility tracking, alpha feedback, and product metrics such as impressions.

Metrics: translation accuracy · expressivity and rhythm preservation · voice similarity · audio/video sync · human rating variance and bias · media eligibility rate · impression lift

Outcomes

• Meta demonstrated AI-generated translated speech and lip-synced video for Reels.
• The feature was in alpha with a small number of creators.
• Meta reported a meaningful increase in impressions because translated media was available in more languages.
• About 90% of submitted media was eligible for translation and lip sync.
• The team identified platformization, latency reduction, monitoring/experimentation integration, expanded language support, music-track support, multi-speaker support, and quality improvements as next steps.

The system coordinated more than ten models, targeted billions of videos, supported six Seamless languages, ran in alpha for a limited creator set, and reported 90% translation/lip-sync eligibility for submitted media.