Video Content Summarization and Metadata Enrichment for Streaming Platform

paramount+ · Media & Entertainment · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Streaming media metadata generation and personalization

Paramount+ needed to automate video summaries and metadata extraction for a large streaming catalog so personalization and discovery could improve without thousands of hours of manual tagging or costly third-party metadata dependence.

Constraints:

Architecture

Paramount+ and Google Cloud built a video-metadata pipeline that stores video assets in cloud storage, demuxes audio, falls back to speech-to-text when transcripts are missing or poor, triggers generation when transcripts land, uses prompt chains and function calling to create summaries and metadata, stores outputs in Firestore by content ID, and feeds personalization systems that join the generated metadata with clickstream data.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cloud-storage-video-assets["Cloud storage video assets"] --> audio-demuxing-pipeline["Audio demuxing pipeline"]
  audio-demuxing-pipeline --> speech-to-text-fallback["Speech-to-text fallback"]
  speech-to-text-fallback --> cloud-storage-video-assets
  cloud-storage-video-assets --> transcript-event-trigger["Transcript event trigger"]
  transcript-event-trigger --> prompt-generation-pipeline["Prompt generation pipeline"]
  external-metadata-services["External metadata services"] --> prompt-generation-pipeline
  prompt-generation-pipeline --> firestore-metadata-store["Firestore metadata store"]
  firestore-metadata-store --> personalization-systems["Personalization systems"]
  personalization-systems --> content-management-systems["Content management systems"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Cloud storage video assets Holds source videos and receives transcript artifacts. Audio demuxing pipeline, Transcript event trigger
Audio demuxing pipeline Extracts the highest-quality audio from HLS video assets. Cloud storage video assets, Speech-to-text fallback
Speech-to-text fallback Creates transcripts with managed Google models or a containerized Whisper distill service when transcripts are missing or low quality. Audio demuxing pipeline, Cloud storage video assets
Transcript event trigger Starts generation workflows when transcript files land in storage. Cloud storage video assets, Prompt generation pipeline
Prompt generation pipeline Uses prompt chaining, few-shot examples, templates, and function calling to generate summaries and metadata. Transcript event trigger, Firestore metadata store, External metadata services
External metadata services Provides structured external data, such as IMDb information, through function calling. Prompt generation pipeline
Firestore metadata store Stores generated key-value outputs by content ID for downstream use. Prompt generation pipeline, Personalization systems
Personalization systems Joins generated metadata with clickstream data and serves discovery/personalization features. Firestore metadata store, Content management systems
Content management systems Downstream backend systems that can consume or update generated metadata. Personalization systems

Data flow

Source videos enter cloud storage; audio is demuxed and transcribed through managed STT or a containerized Whisper fallback; transcript-arrival events trigger prompt-driven generation; generated summaries and metadata are stored as Firestore key-value records keyed by content ID; personalization systems combine those records with clickstream data and downstream content-management workflows.

3 supporting quotes
“The reference architecture consists of three main components: transcription creation, generation phase, and personalization integration.”summary, stated
“The generation phase is triggered by events when transcripts land in storage buckets.”summary, stated
“The generation process stores outputs in Firestore as key-value pairs (with content ID as the key), making them available for updates within the personalization system.”summary, stated

Why this architecture

The pipeline architecture fits the workload because video summarization depends on transcript availability, metadata needs to flow into personalization systems by content ID, and prompt chains give the team control over long-form generation, genre-specific outputs, and future feedback-driven tuning.

“Metadata serves a dual role in their architecture: powering machine learning algorithms that fuel recommendation systems, and providing viewers with the content information they need for discovery.”summary, stated
“• Prompt Chaining: Breaking tasks into subtasks allows better control over the context window, improves debugging of production processes, and reduces downstream workload”summary, stated

Models used: Google LLMs · Whisper distill v3 · Gemma 2B instruct · T5 · LoRA fine-tuned embedding models · Encoder-only classifiers

Scale

Alternatives considered (roads not taken)

Continue manual summaries and third-party metadata procurement

Rejected because: Manual tagging consumed thousands of hours and third-party metadata was costly and insufficiently detailed for personalization.

“The practical motivation for these AI projects is substantial. Paramount+ has over 50,000 videos requiring summaries and metadata tagging, which translates into thousands of hours of manual labor.”summary, stated

Depend only on managed Google speech-to-text

Rejected because: The transcription architecture explicitly kept open-source Whisper distill as a fallback service.

“Paramount maintains flexibility in their model choices—they can use Google managed service models or open-source alternatives. They specifically mentioned running a containerized Whisper distill version 3 as its own service”summary, stated

Use a single generic prompt for all content

Rejected because: The roadmap moves toward genre-specific prompt templates, audience affinities, and feedback-driven scoring.

“They developed genre-specific or micro-genre classified prompts, with plans to associate different summaries with different audience affinities.”summary, stated

Patterns

workflow-orchestration (pipeline)

Transcript storage events trigger a multi-stage generation workflow that moves from transcription to summary/metadata generation and personalization integration.

“The generation phase is triggered by events when transcripts land in storage buckets. The generation process stores outputs in Firestore as key-value pairs (with content ID as the key), making them available for updates within the personalization system.”summary, stated

context-engineering (prompting)

The team engineered prompt context through few-shot examples, templates, output formats, genre prompts, and prompt chaining to make long-video metadata generation controllable.

“The team emphasized that LLM response effectiveness depends heavily on well-detailed, clear prompts with appropriate constraints.”summary, stated
“• Prompt Chaining: Breaking tasks into subtasks allows better control over the context window, improves debugging of production processes, and reduces downstream workload”summary, stated

model-fallback (model operations)

The transcription system can fall back to speech-to-text models, including a containerized Whisper service, when transcripts are absent or poor.

“The speech-to-text (STT) process serves as a fallback method when transcripts don't exist or are of insufficient quality.”summary, stated

eval-driven-development (quality)

The engagement used gold standards, daily feedback, prompt iterations, and planned summary scoring/A-B testing to improve generated metadata quality.

“Daily communication and in-depth working sessions enabled gathering feedback early and often. Getting access to videos and expected metadata output before the engagement began allowed them to "hit the ground running" with a clear north star for the solution.”summary, stated
“Candidate Selection Strategy: Using encoder-only classifiers to inject estimated scores for summaries, enabling A/B testing of generated content.”summary, stated

fine-tuning-feedback-loop (model improvement)

The roadmap includes human preference datasets, reward models, and scheduled fine-tuning updates for generated summaries.

“The architecture supports continuous feedback loops essential for adaptive personalization systems. On Google Cloud, the reinforcement learning fine-tuning process is supported by model types like T5 and Gemma. The workflow involves:”summary, stated
“• Reward Model Training: Human preference data trains reward models, updating model weights that are stored for the fine-tuned model.”summary, stated

Tradeoffs

Keep both managed and open-source transcription options

Chose Google managed STT plus a containerized Whisper distill service over A single vendor transcription path.

Rationale: The team wanted flexibility when transcripts were missing or insufficient.

“Paramount maintains flexibility in their model choices—they can use Google managed service models or open-source alternatives.”summary, stated

Use prompt chaining for long, debuggable workflows

Chose Subtask prompts and chained generation over One large monolithic prompt.

Rationale: Prompt chaining controls context windows, simplifies debugging, and reduces downstream workload.

“• Prompt Chaining: Breaking tasks into subtasks allows better control over the context window, improves debugging of production processes, and reduces downstream workload”summary, stated

Fine-tune embeddings to shrink model and embedding requirements

Chose LoRA fine-tuning and smaller models on single GPUs over Larger general embedding spaces and heavier models.

Rationale: The team wanted information-dense embeddings that fit their operational GPU constraints.

“They employ LoRA (Low-Rank Adaptation) fine-tuning to shrink the embedding space to their required size. The rationale is creating the most information-dense space possible while enabling smaller models to run on single GPUs.”summary, stated
“The rationale is creating the most information-dense space possible while enabling smaller models to run on single GPUs.”summary, stated

Gotchas (what bit them)

Long movie transcripts can exceed context windows.

Root cause: Full movies produce transcripts too large for a single prompt or model context.

Resolution: Chunk transcripts and aggregate results after generation.

“Token Limits: Movie transcripts can be quite long, potentially exceeding token limits. The solution involves chunking transcripts and aggregating results at the end.”summary, stated

Mature entertainment content can trigger safety filters.

Root cause: Movies and shows may contain mature themes that general-purpose safety settings suppress.

Resolution: Tune safety filter settings to allow appropriate outputs for the entertainment domain.

“• Safety Filters: Movies and shows with mature themes required addressing safety filter settings to allow appropriate outputs—a common challenge in entertainment content”summary, stated

Prompt quality required repeated iteration.

Root cause: Expected metadata format, genre nuance, and summary style had to be clarified through examples and feedback.

Resolution: Use daily communication, gold standards, few-shot examples, and four MVP prompt iterations.

“The engagement required four different iterations to refine the MVP prompt to meet Paramount's expectations. Daily communication and in-depth working sessions enabled gathering feedback early and often.”summary, stated
“• Gold Standards: Establishing clear standards for generated output before beginning work (e.g., whether the goal is a generalized plot summary or a summary with a specific voice or point of view) keeps development on target.”summary, stated

Vendor event summaries do not prove business impact.

Root cause: The presentation gives scale and approach but not measured cost savings, engagement lift, or production outcome metrics.

Resolution: Treat the case as an architecture and process lesson rather than a fully quantified ROI proof.

“The specific quantitative results (cost savings, time savings, engagement improvements) are not provided, making it difficult to assess the actual impact beyond validation of technical hypotheses.”summary, stated

Evaluation

The engagement evaluated prompt and metadata quality through gold standards, daily feedback, four MVP prompt iterations, and a roadmap for classifier scoring, A/B tests, human preference datasets, reward models, and reinforcement-learning fine-tuning.

Metrics: Four prompt iterations · 10 representative video assets for the Jump Start · Future summary scoring classifiers · A/B testing of generated content · Human preference datasets for reward-model training

3 supporting quotes
“Getting access to videos and expected metadata output before the engagement began allowed them to "hit the ground running" with a clear north star for the solution.”summary, stated
“• Human Preference Datasets: For summaries, this might be a random selection of summaries where people have determined accuracy and interest-eliciting qualities”summary, stated
“Candidate Selection Strategy: Using encoder-only classifiers to inject estimated scores for summaries, enabling A/B testing of generated content.”summary, stated

Outcomes

The case addresses a catalog of more than 50,000 videos and started from 10 representative assets during the GenAI Jump Start.

“Results validated their hypothesis that clear context with specific examples improves output quality, and established a roadmap for genre-specific prompt templates, summary scoring classifiers for A/B testing, and reinforcement learning fine-tuning.”summary, stated
“The technical architecture described is sound and reflects production-ready thinking, with appropriate attention to fallback mechanisms (STT when transcripts are unavailable), event-driven processing, and integration with existing systems.”summary, stated