Large Recommender Models: Adapting Gemini for YouTube Video Recommendations

google_/_youtube · Media & Entertainment · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Video recommendation at consumer-internet scale

YouTube wanted to adapt Gemini-style large language modeling to video recommendations, where the model must reason over users, context, watch history, and billions of videos. The core challenge was converting videos into tokens a language model could use, adapting Gemini to YouTube's domain language, and serving improved recommendations under YouTube-scale cost, latency, freshness, and personalization constraints.

Constraints:

• YouTube recommendation surfaces include home, watch next, Shorts, and personalized search results
• the system must model user demographics, location, device, recent watch history, engagement depth, comments, and subscriptions
• video representations must be compressed enough for LLM context and decoding
• the system operates over billions of videos with millions added every day
• fresh content needs to be understood and recommended within minutes or hours
• Gemini Pro-class models cannot be directly served to billions of daily active users
• recommendation models must hit strict serving cost and latency requirements

Architecture

YouTube builds a Large Recommender Model by adapting a Gemini checkpoint to a YouTube-specific video language. Videos are converted into semantic IDs by extracting title, description, transcript, audio, and frame-level features, embedding them, and quantizing them with RQ-VAE-like residual vector quantization. Continued pretraining links English text to semantic ID tokens and trains on masked watch-sequence prediction over YouTube engagement data. The resulting LRM can generate video semantic IDs for recommendation tasks such as retrieval, and production deployment uses smaller checkpoints plus cost optimizations and offline recommendation tables to avoid per-user online inference where possible.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  corpus["YouTube video corpus"] --> tokenizer["Semantic ID tokenizer"]
  gemini["Base Gemini checkpoint"] --> lrm["YouTube LRM checkpoint"]
  tokenizer --> lrm
  watches["Watch-sequence training data"] --> lrm
  lrm --> prompt["Generative retrieval prompt"]
  prompt --> surface["Surface-specific retrieval and ranking models"]
  lrm --> surface
  surface --> table["Offline recommendation table"]
  table --> prod["Production recommendation surfaces"]
  surface --> prod

Components

Data flow

YouTube starts with a base Gemini checkpoint and adapts it into a YouTube LRM checkpoint. Each video is featurized from metadata, transcripts, audio, and video frames, embedded, and quantized into semantic ID tokens. Continued pretraining teaches the model to map between English and semantic IDs and to predict masked videos from user watch sequences. For generative retrieval, a user prompt includes demographics, current context, and recent watch history, and the model decodes recommended video semantic IDs. For lower-cost serving, YouTube also removes personalization from some prompts, performs offline inference over head videos, builds recommendation lookup tables, and serves candidates through simple lookup.

Why this architecture

YouTube uses semantic IDs because raw video content is too large and heterogeneous for direct LLM tokenization. Continued pretraining turns Gemini into a bilingual model over English and YouTube's video-token language, letting it reason about relationships between videos and users. Serving is split between smaller online models and offline lookup tables because full Gemini-scale inference is too costly for billions of users.

Models used: Gemini · Large Recommender Model · Gemini Flash · smaller Gemini checkpoints · semantic ID model · RQ-VAE or residual quantization vector quantization · generative retrieval model · mixture-of-experts candidates

Scale

• YouTube is described as one of the largest consumer apps in the world.
  “It's one of the biggest consumer apps in the world.”transcript, stated
• A large majority of YouTube watch time is driven by the recommendation system.
  “a large majority of the watch time on YouTube is driven by the recommendation system.”transcript, stated
• The model consumes the last 100 watched videos as part of user context.
  “What are the last 100 videos they watched? How deeply did they engage with them?”transcript, stated
• The speaker says YouTube has 20 billion videos with millions added every day.
  “We have 20 billion videos on YouTube, um with millions added every day.”transcript, stated
• Fresh videos need recommendation understanding within minutes or hours.
  “you have to be able to recommend it within the next minutes or hours, otherwise a lot of users are going to be upset.”transcript, stated
• The team achieved more than 95% TPU serving-cost savings before production launch.
  “we got 95% plus cost savings to be able to actually launch this in production.”transcript, stated
• The speaker contrasts YouTube continuous pretraining on days/hours with Gemini-like cycles of three to six months.
  “continuously pre-train it on the order of days and hours which is very different than classical LLM pre-training like Gemini which happens maybe like once in 3 to 6 months.”transcript, stated

Alternatives considered (roads not taken)

Predict search queries and retrieve videos through those queries

Rejected because: The speaker says this was tried before semantic IDs and was not good enough.

Use hash-based tokenization for videos

Rejected because: YouTube wanted semantically meaningful tokens that preserve relationships between videos.

Serve the strongest Gemini Pro-class model to all YouTube users

Rejected because: The full model was not servable to billions of daily active users under YouTube latency and scale requirements.

Always serve personalized LRM recommendations online

Rejected because: The team converted some recommendation serving into offline lookup tables to avoid serving-cost bottlenecks.

Patterns

retrieval-ranking (recommendation)

The LRM is aligned for recommendation retrieval and ranking surfaces, generating candidate video semantic IDs for downstream recommendation systems.

fine-tuning-adaptation (model-adaptation)

YouTube starts from Gemini and continues pretraining it on YouTube-specific semantic-ID and watch-sequence tasks.

batch-offline-processing (serving)

To reduce online serving cost, YouTube precomputes recommendations for head videos and serves them through lookup tables.

latency-optimization (serving)

The team moved toward smaller Gemini Flash and even smaller checkpoints because the recommendation system must meet latency and scale requirements.

cost-aware-routing (serving)

The production design routes some recommendation work to offline precomputed tables and smaller models after 95%+ TPU serving-cost reduction work.

Tradeoffs

How to let Gemini reason over videos

Chose Semantic ID tokenization over Search-query prediction, direct video recommendation, or hash-based video tokens.

Rationale: Semantic IDs compress videos into meaningful tokens that preserve relationships and outperformed the earlier retrieval approaches the team tried.

How to adapt Gemini for YouTube

Chose Continued pretraining that links English and semantic IDs plus watch-sequence reasoning over Using Gemini as a generic text-only recommender.

Rationale: The adaptation creates a bilingual model that can speak natural language and YouTube's domain-specific video language.

How to serve recommendations at YouTube scale

Chose Smaller checkpoints, cost reduction, and offline recommendation tables over Full personalized LRM online inference for every request.

Rationale: The LRM improved quality but serving cost and latency required aggressive optimization and offline lookup.

How to handle language capability while specializing on video tokens

Chose Balance semantic-ID expertise with retained text ability over Overtraining solely on semantic-ID recommendation quality.

Rationale: Future interactive recommendation features require natural-language input and explanations, so losing English ability is a problem.

Gotchas (what bit them)

Raw video content cannot be treated like ordinary text tokens.

Root cause: The model needs to reason over many videos, but even huge context windows require compressed video representation.

Resolution: Create semantic IDs by embedding multimodal video features and quantizing them into video tokens.

High-quality LRM retrieval was initially too expensive to serve.

Root cause: Transformer-based recommendation serving at YouTube's billion-user scale creates high TPU serving cost.

Resolution: Reduce serving cost by more than 95%, use smaller checkpoints, and turn some personalized serving into offline lookup.

Recommendation freshness requirements are much stricter than ordinary LLM pretraining cadence.

Root cause: YouTube adds millions of videos daily and users expect trending fresh content immediately.

Resolution: Continuously pretrain the LRM on days/hours timescales rather than multi-month foundation-model cycles.

Overtraining on semantic-ID tasks can damage the model's English capability.

Root cause: The model can over-index on the new video-token language and forget natural-language behavior.

Resolution: Explore balancing mechanisms such as mixture-of-experts, with some experts retaining text ability while others focus on semantic IDs.

Video-frame features are heavily compressed and may obscure what the representation captured.

Root cause: At YouTube scale, only selected frames can be sampled and all video information is compressed into a small token representation.

Resolution: Sample frames intelligently, including from high-engagement moments, while acknowledging uncertainty in exactly what the token picked up.

Evaluation

The talk describes production launch for retrieval, experimentation for ranking, comparison against prior production recommendations in hard cases, cold-start/fresh-content improvements, and internal quality assessment. Specific quantitative quality metrics are withheld, but the speaker reports that it was the largest recommendation-quality improvement in recent years.

Metrics: recommendation quality · retrieval production launch status · ranking experiment status · TPU serving cost · fresh-video performance · new-user and limited-history recommendation quality · cold-start performance for videos uploaded in the last day or week

Outcomes

• YouTube launched LRM in production for retrieval and is experimenting with it for ranking.
• The system produced unique recommendations in hard cases such as users with limited information and Olympic-watch scenarios.
• The team achieved more than 95% TPU serving-cost savings before launch.
• Offline recommendation tables provided a workaround for serving-cost bottlenecks on popular videos.
• Semantic IDs improved fresh and tail-content performance for videos uploaded in the last day or week.
• The speaker says the system is the largest recommendation-quality improvement YouTube has seen in the last few years, while withholding exact metrics.

The system is designed for one of the world's largest consumer apps, across home, watch-next, Shorts, and search surfaces, with 20 billion videos, millions of daily uploads, billions of daily active users in the serving constraint statement, and continuous adaptation on days/hours timescales.