AI-Powered Content Moderation at Platform Scale

roblox · Media & Entertainment · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: User-generated-content safety

Roblox needed to moderate billions of daily messages, voice interactions, and uploaded assets in real time across many languages while maintaining policy quality and user trust.

Constraints:

• Moderation must happen within milliseconds or seconds for live communication
• Human-only review cannot scale to Roblox traffic
• Language, slang, memes, and evasion tactics continually change

Architecture

Roblox runs a multi-modal moderation platform where purpose-built transformer classifiers, GPU serving, quantization/distillation, voice safety models, real-time warnings, human experts, active-learning datasets, appeals, abuse reports, synthetic examples, and AI-assisted red teams continuously improve safety decisions across text, voice, images, and assets at massive scale.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  text-filtering-models["text-filtering-models"]
  gpu-serving-stack["gpu-serving-stack"]
  voice-safety-classifier["voice-safety-classifier"]
  real-time-feedback-system["real-time-feedback-system"]
  human-expert-oversight["human-expert-oversight"]
  training-data-pipeline["training-data-pipeline"]
  text-filtering-models --> gpu-serving-stack
  text-filtering-models --> real-time-feedback-system
  text-filtering-models --> training-data-pipeline
  voice-safety-classifier --> real-time-feedback-system
  voice-safety-classifier --> training-data-pipeline
  human-expert-oversight --> training-data-pipeline
  training-data-pipeline --> text-filtering-models
  training-data-pipeline --> voice-safety-classifier

Components

Data flow

User content enters moderation before or during exposure. Text messages become filtering requests routed through purpose-built models on optimized serving stacks; voice is assessed within seconds and returned as user-facing warnings rather than pre-blocking; human experts, appeals, abuse reports, golden sets, uncertainty sampling, AI-assisted red teams, and synthetic data feed new examples back into training and evaluation datasets.

Why this architecture

Roblox needs layered ML plus human oversight because human-only moderation cannot operate at millisecond latency or billion-item scale, while rare/evolving cases still require expert judgment. The serving architecture optimizes throughput without treating quality and scale as opposites: distillation and quantization improved capacity while reducing false positives and increasing detection.

Models used: large transformer-based models · PII filter model · voice safety classifier · LLMs for synthetic data

Scale

• Roblox averages 97.8 million daily active users.
  “An average of 97.8 million daily active users_1_ come to Roblox to play, communicate, and create.”article, stated
• Users send an average of 6.1 billion chat messages and 1.1 million hours of voice communication per day in 28 languages.
  “Every day, users send an average of 6.1 billion chat messages and 1.1 million hours of voice communication in 28 different languages.”article, stated
• From February to December 2024, users uploaded about 1 trillion pieces of content, with about 0.01% violating policies.
  “From February to December, 2024_1_, users uploaded approximately 1 trillion pieces of content. As little as 0.01% of those billions of those text chats, audio, voice, and images were detected as violating any of our policies.”article, stated
• The PII filter handles 370,000 RPS at peak after GPU-stack optimization.
  “On the new stack, the PII filter is now handling 370,000 RPS at peak.”article, stated
• Text filters collectively handle more than 750,000 RPS.
  “the ones managing our text filters are now efficiently handling more than 750,000 RPS.”article, stated
• The voice classifier handles up to 8,300 RPS at peak and works within 15 seconds across eight languages.
  “moderates chat within 15 seconds across eight languages.”article, stated

  “and is handling, at peak, up to 8,300 RPS.”article, stated

Alternatives considered (roads not taken)

Human-only moderation at Roblox scale

Rejected because: The article says millisecond moderation at this volume would require impractical numbers of human reviewers and still miss the speed requirement.

Deploying AI whenever it is faster

Rejected because: Roblox only deploys AI when it exceeds human precision and recall at scale, keeping humans for nuanced and evolving cases.

Blocking voice pre-emptively like text

Rejected because: Voice communication cannot be blocked in the same way, so Roblox uses rapid classifier feedback and warnings.

Patterns

safety-policy-enforcement (content-moderation)

Roblox blocks policy-violating text in real time and applies escalating consequences for repeated violations.

human-in-the-loop-gate (moderation-oversight)

Humans handle nuanced, rare, evolving, investigative, and appeals cases while also training moderation systems.

latency-optimization (model-serving)

Roblox moved high-RPS PII filtering to GPUs, separated tokenization from inference, and used quantization/distillation to increase throughput.

data-quality-loop (moderation-training)

Roblox continuously curates and evaluates training data using golden sets, uncertainty sampling, human experts, appeals, and abuse-report context.

synthetic-data-generation (rare-case-coverage)

LLMs generate artificial examples and labels to cover rare or edge-case moderation scenarios where real labeled data is insufficient.

Tradeoffs

How to balance false negatives and false positives.

Chose optimize for fewer false negatives while continuously reducing false positives over maximizing permissiveness or relying only on broad removal.

Rationale: Safety requires catching violations, but excessive compliant-content removals frustrate users and require ongoing quality loops.

When to use AI for moderation decisions.

Chose deploy AI only when it exceeds human precision and recall at scale over deploying AI merely because it is faster.

Rationale: Roblox treats AI speed as insufficient without quality that beats human-scale moderation performance.

Gotchas (what bit them)

Human moderation alone cannot keep up with real-time scale.

Root cause: Billions of chat, voice, image, and asset events require millisecond decisions across global traffic.

Resolution: Roblox uses scalable ML infrastructure plus human experts for oversight and hard cases.

CPU serving became insufficient for the PII filter.

Root cause: Every chat message is a moderation request, driving very high RPS on a policy-critical model.

Resolution: Roblox built a GPU serving stack and optimized inference with tokenization separation, quantization, and distillation.

Training data can look good offline but fail in production.

Root cause: Easy evaluation sets and inaccurate labels hide rare, evolving, or policy-confusing cases.

Resolution: Roblox evaluates alignment and quality, samples edge cases, and iterates policies/training when humans disagree.

Evaluation

Roblox evaluates moderation systems with precision/recall, false-positive rates, RPS, alignment among human labels, golden-set quality, expert review, and production behavior changes.

Metrics: precision · recall · false positive rate · RPS · alignment >=80% · golden-set quality · filtered-message reduction

Outcomes

• The PII filter handles 370,000 RPS at peak.
• PII false positives fell by 30%.
• Text filters handle more than 750,000 RPS.
• The voice classifier has 92% higher recall than the initial version with a 1% false positive rate.
• Text-chat notifications/time-outs reduced filtered messages by 5% and abuse-report consequences by 6% in experiments.

Roblox reports 97.8 million daily active users, 6.1 billion chat messages per day, 1.1 million hours of daily voice communication, and approximately 1 trillion uploaded pieces of content from February through December 2024.