Video Super-Resolution at Scale for Ads and Generative AI Content

meta · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale video enhancement and AI model serving

Meta needed to improve the quality of low-resolution or degraded video assets across a massive video ecosystem without exhausting scarce GPU resources, creating artifacts, or changing creator intent.

Constraints:

Architecture

Meta deployed video super-resolution as a portfolio of server-side and GPU-side enhancement paths: ingestion-side VSR creates higher-quality source variants for ABR encoding, CPU VSR uses Intel RVSR/OpenVINO/FFmpeg for cost-effective processing, GPU VSR supports latency-sensitive Restyle and MovieGen post-processing, a multi-variant content model manages enhanced versions, and subjective evaluation plus VMAF-UQ targeting decides which videos should receive expensive enhancement.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  video-upload-and-inventory["Video upload and inventory"] --> quality-targeting-layer["Quality targeting layer"]
  quality-targeting-layer --> cpu-vsr-pipeline["CPU VSR pipeline"]
  quality-targeting-layer --> gpu-vsr-pipeline["GPU VSR pipeline"]
  gpu-vsr-pipeline --> restyle-and-moviegen-pipeline["Restyle and MovieGen pipeline"]
  cpu-vsr-pipeline --> multi-variant-video-model["Multi-variant video model"]
  gpu-vsr-pipeline --> multi-variant-video-model
  multi-variant-video-model --> abr-encoding-and-delivery["ABR encoding and delivery"]
  subjective-evaluation-framework["Subjective evaluation framework"] --> quality-targeting-layer
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Video upload and inventory Receives user uploads, legacy assets, ads inventory, and generated video content. Quality targeting layer
Quality targeting layer Uses quality metrics and evaluation findings to choose which videos are worth enhancing. Video upload and inventory, CPU VSR pipeline, GPU VSR pipeline
CPU VSR pipeline Runs optimized VSR through Intel RVSR, OpenVINO, and FFmpeg plugins for cost-efficient server processing. Quality targeting layer, Multi-variant video model
GPU VSR pipeline Runs latency-sensitive VSR for Restyle and MovieGen-related processing, including parallel segment processing. Quality targeting layer, Restyle and MovieGen pipeline, Multi-variant video model
Restyle and MovieGen pipeline Transforms user media with generative AI and optional post-processing such as frame-rate upconversion and super-resolution. GPU VSR pipeline
Multi-variant video model Stores enhanced variants as versions of the same video for rollout, encoding, and delivery decisions. CPU VSR pipeline, GPU VSR pipeline, ABR encoding and delivery
ABR encoding and delivery Encodes and serves selected video variants to users. Multi-variant video model
Subjective evaluation framework Collects crowd ratings, preferences, artifact labels, and statistical analysis for VSR quality. Quality targeting layer

Data flow

Uploaded or generated videos are assessed for quality; eligible lower-resolution assets are enhanced through CPU or GPU VSR depending on latency and resource needs; enhanced variants are stored as versions of the same video object; ABR encoding and delivery can serve the chosen variant; and subjective evaluation plus VMAF-UQ analysis guide targeting and rollout.

3 supporting quotes
“On the ingestion side, when receiving lower-resolution, lower-quality videos, they apply VSR to upscale content and create high-quality source material for downstream adaptive bitrate (ABR) encoding.”summary, stated
“To support VSR deployment at scale, Meta expanded their video data model to support multi-variant content.”summary, stated
“Meta built an automated framework for large-scale crowd-based subjective evaluation.”summary, stated

Why this architecture

Meta combined CPU and GPU serving paths because different workloads have different economics: batch enhancement can use abundant CPU infrastructure, latency-sensitive generative video features justify GPU processing, and targeting avoids spending compute on videos where VSR does not improve perceived quality.

“By maintaining a portfolio of solutions that can run on both CPUs and GPUs, Meta gains operational flexibility and cost efficiency.”summary, stated
“This finding has significant operational implications: by targeting VSR only to specific videos that can benefit from it, Meta can substantially reduce overall compute costs while maintaining the same perceived quality improvements.”summary, stated

Models used: Video super-resolution models · Intel RVSR SDK models · OpenVINO-optimized models · MovieGen · VMAF-UQ quality metric

Scale

Alternatives considered (roads not taken)

Rely solely on GPU VSR

Rejected because: GPU resources are scarce and expensive, so Meta also deployed CPU-based VSR for suitable workloads.

“Rather than relying solely on scarce GPU resources—which are in high demand across Meta for various AI-powered features—the team partnered with Intel to deploy advanced VSR models on standard x86 CPU infrastructure.”summary, stated

Apply VSR to every video indiscriminately

Rejected because: Evaluation showed very-low-quality videos do not meaningfully benefit, so targeting saves compute without losing perceived quality.

“If the input video quality is already very low, applying VSR shows no noticeable improvement in subjective ratings.”summary, stated

Use automated metrics alone

Rejected because: VSR lacks universally accepted metrics, and human evaluation catches subjective preferences and artifacts.

“VSR lacks universally accepted quality metrics that can reliably measure quality improvement or detect introduced artifacts.”summary, stated

Patterns

eval-driven-development (evaluation)

Meta built large-scale subjective evaluation and used human ratings plus statistical analysis to choose VSR strategies and proxy metrics.

“Meta built an automated framework for large-scale crowd-based subjective evaluation. The framework displays videos processed by different VSR solutions side-by-side and asks human raters to provide MOS (Mean Opinion Score) ratings for each video and indicate their preferences.”summary, stated
“After collecting raw ratings, Meta leverages state-of-the-art statistical methodology to analyze the data and extract insights.”summary, stated

model-routing (serving)

Different VSR implementations are routed to CPU or GPU infrastructure depending on workload latency and resource constraints.

“Different use cases can be matched to appropriate infrastructure based on their requirements. Latency-sensitive applications like Restyle use GPU-based processing, while batch processing of ad inventory can leverage CPU-based solutions.”summary, stated

cost-aware-serving (infrastructure)

Meta targets VSR only to videos likely to benefit and uses CPU infrastructure where latency does not require GPUs.

“The discovery that only medium-to-high quality input videos benefit from VSR enables targeted deployment that optimizes this tradeoff.”summary, stated
“CPUs are abundant and cost-effective but may have higher latency or require more optimized models to achieve acceptable performance.”summary, stated

human-in-the-loop-gate (quality assurance)

Human raters evaluate side-by-side VSR outputs and identify artifacts that automated metrics may miss.

“The framework displays videos processed by different VSR solutions side-by-side and asks human raters to provide MOS (Mean Opinion Score) ratings for each video and indicate their preferences.”summary, stated

Tradeoffs

Maintain CPU and GPU VSR paths

Chose A portfolio of CPU-based and GPU-based VSR capabilities over A single infrastructure path.

Rationale: CPU processing offers cost and availability advantages, while GPUs are reserved for latency-sensitive or high-performance workloads.

“Meta's solution—maintaining both CPU-based and GPU-based VSR capabilities—provides flexibility but also increases system complexity.”summary, stated

Target VSR based on input quality

Chose Selective enhancement for videos likely to benefit over Enhancing every video.

Rationale: Human evaluation showed that very-low-quality inputs did not improve subjectively, so targeting saves compute.

“Rather than applying expensive processing to all videos indiscriminately, Meta can intelligently select which assets to enhance based on input quality metrics.”summary, stated

Use multi-variant video records

Chose Enhanced variants attached to the same video object over Creating entirely separate video objects for every enhancement.

Rationale: Variants support efficient management, rollout, A/B testing, storage optimization, and delivery decisions.

“Rather than proliferating entirely separate video objects, the variant system allows for efficient management, storage optimization, and flexible delivery decisions.”summary, stated

Gotchas (what bit them)

Automated metrics alone were not enough for VSR quality.

Root cause: VSR does not have universally accepted metrics that reliably capture subjective quality improvement and artifacts.

Resolution: Build crowd-based subjective evaluation and use VMAF-UQ only after validating correlation with human ratings.

“Unlike video compression, which has well-established quality metrics and methodologies to benchmark different codecs (like PSNR, SSIM, and VMAF), VSR lacks universally accepted quality metrics”summary, stated
“First, they found that VMAF-UQ (a variant of VMAF developed by Google) shows very good correlation with human subjective ratings.”summary, stated

Very-low-quality inputs did not benefit from enhancement.

Root cause: VSR could not recover meaningful perceived quality when source quality was already too poor.

Resolution: Target enhancement to medium-to-high quality input videos where subjective ratings improve.

“When grouping videos based on their input VMAF-UQ scores, they discovered that only videos with medium to high quality can meaningfully benefit from VSR.”summary, stated

Mobile client-side VSR remained difficult.

Root cause: Sophisticated VSR models exceed typical mobile power and compute budgets.

Resolution: Focus primarily on server-side deployment while continuing mobile-focused research.

“deploying sophisticated VSR models on typical mobile platforms remains "very challenging" due to power and compute limitations.”summary, stated

Parallel GPU processing reduces latency but uses more scarce resources.

Root cause: Restyle-like user-facing workloads require low end-to-end latency, unlike batch enhancement.

Resolution: Split videos into segments and process them across GPUs only for latency-sensitive features.

“The parallel processing approach for Restyle—splitting videos into segments and processing them on multiple GPUs simultaneously—represents a classic scaling pattern that trades increased resource utilization for reduced latency.”summary, stated

Evaluation

Meta evaluated VSR with side-by-side crowd ratings, MOS preference collection, artifact labels, statistical analysis, and correlation analysis against VMAF-UQ to find a scalable proxy metric.

Metrics: MOS ratings · Rater preference comparisons · Artifact labels · VMAF-UQ correlation with subjective ratings · Input-quality buckets for targeted deployment

3 supporting quotes
“The framework displays videos processed by different VSR solutions side-by-side and asks human raters to provide MOS (Mean Opinion Score) ratings for each video and indicate their preferences.”summary, stated
“Critically, raters are also asked to identify any artifacts they observe in the videos, enabling detection of failure modes that automated metrics might miss.”summary, stated
“The presentation includes charts comparing MOS improvement with different VSR solutions alongside corresponding VMAF-UQ scores, demonstrating strong alignment.”summary, stated

Outcomes

The deployment operates in an ecosystem with over 1 billion daily uploads, more than 1 trillion view requests, and more than 1 trillion video assets.

“Meta deployed VSR technology for two primary use cases during this initiative. The first was enhancing lower-resolution advertising videos and images in their inventory.”summary, stated
“This kind of conditional deployment—applying expensive AI processing only where it provides measurable benefit—is a key optimization strategy in production AI systems.”summary, stated