AI-Powered Artwork Quality Moderation and Streaming Quality Management at Scale

amazon_prime_video · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Streaming media quality operations

Prime Video needed to scale artwork-quality moderation and streaming-quality incident response across a global video platform with many titles, partners, viewers, languages, and live events.

Constraints:

Architecture

Prime Video described two AI operations patterns: multimodal artwork moderation broken into defect-specific loops with standardized ground truth, and a streaming-quality agent workflow with curated context, guardrails, mitigation levers, and decision logs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  artwork-moderation-workflow["Artwork moderation workflow"]
  defect-type-loop["Defect-type loop"]
  manual-ground-truth["Manual ground truth"]
  evaluation-loop["Evaluation loop"]
  streaming-quality-request-handler["Streaming quality request handler"]
  context-curation["Context curation"]
  guardrail-agent["Guardrail agent"]
  root-cause-and-mitigation-agents["Root-cause and mitigation agents"]
  decision-logs["Decision logs"]
  human-involvement["Human involvement"]
  artwork-moderation-workflow --> defect-type-loop
  artwork-moderation-workflow --> manual-ground-truth
  manual-ground-truth --> evaluation-loop
  defect-type-loop --> evaluation-loop
  streaming-quality-request-handler --> guardrail-agent
  streaming-quality-request-handler --> context-curation
  context-curation --> root-cause-and-mitigation-agents
  guardrail-agent --> root-cause-and-mitigation-agents
  root-cause-and-mitigation-agents --> decision-logs
  decision-logs --> evaluation-loop
  guardrail-agent --> human-involvement
  root-cause-and-mitigation-agents --> human-involvement
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Artwork moderation workflow Runs multimodal defect checks over artwork assets. Defect-type loop, Manual ground truth
Defect-type loop Breaks moderation into individual defect types ranked by frequency and effort. Evaluation loop
Manual ground truth Standardizes evaluator criteria and supplies labels for automated checks. Evaluation loop
Evaluation loop Measures false positives, false negatives, time to result, and manual effort reduction. Artwork moderation workflow
Streaming quality request handler Starts incident workflows for streaming-quality issues. Guardrail agent, Context curation
Context curation Selects high-quality operational data before agent analysis. Root-cause and mitigation agents
Guardrail agent Limits unsafe actions and routes uncertain cases to human involvement. Root-cause and mitigation agents, Human involvement
Root-cause and mitigation agents Detect, localize, root-cause, and propose mitigations for streaming issues. Decision logs, Human involvement
Decision logs Capture actions and evidence for reflective analysis and future improvement. Evaluation loop
Human involvement Handles safe failure modes and review points. Guardrail agent

Data flow

Artwork defects are handled one type at a time through evaluation loops, while streaming quality incidents use agentic workflows to detect, localize, root-cause, mitigate, and log decisions.

“building an agentic workflow to detect localize root cause and mitigate streaming quality issues.”transcript, stated
“we broke the problem down into the individual defect types, ranked them by how often they occurred and how much effort it was to manually perform the detection and tackle them one at a time.”transcript, implied

Why this architecture

The architecture invests in standardized evaluator criteria and curated operational context because inconsistent labels and too much noisy data hurt automated quality systems.

“we ended up creating a standard operating procedure for the manual evaluators that would also be shared with the automated system and so we could have consistent results.”transcript, stated
“data quality always beats data quantity when you're trying to develop such a system”transcript, stated

Models used: multimodal foundation models · generative AI agents

Scale

Alternatives considered (roads not taken)

Detect all artwork defect types in one pass

Rejected because: The team found broad simultaneous detection performed poorly and instead ranked and tackled defect types individually.

“we initially tried to take advantage of that. We tried to run defect detection for multiple types of issues at once and found that that was just not the way to go.”transcript, stated
“we broke the problem down into the individual defect types, ranked them by how often they occurred and how much effort it was to manually perform the detection and tackle them one at a time.”transcript, stated

Fully autonomous streaming mitigation without review

Rejected because: The system needs safe failure modes and human involvement when autonomy is inappropriate.

“we want to have an autonomous system but there are going to be times when you might want to have sort of human evaluation”transcript, stated

Inconsistent human criteria as the evaluation baseline

Rejected because: Prime Video created a standard operating procedure shared by evaluators and the automated system.

“we ended up creating a standard operating procedure for the manual evaluators that would also be shared with the automated system and so we could have consistent results.”transcript, stated

Feeding all available operational data to the incident agent

Rejected because: The team found data quality beats data quantity for this agentic streaming-quality system.

“data quality always beats data quantity when you're trying to develop such a system”transcript, stated

Patterns

multi-agent-orchestration (orchestration)

The streaming-quality system is described as an agentic workflow that coordinates specialized steps for detection, localization, root cause, response handling, and guardrails.

“building an agentic workflow to detect localize root cause and mitigate streaming quality issues.”transcript, stated

eval-driven-development (evaluation)

Artwork moderation improvements are driven by repeated evaluation against human-labeled ground truth and analysis of false positives and false negatives.

“We were able to reduce false positives and negatives by 70% and we were also able to reduce the amount of time it takes to get results from several days to less than an hour”transcript, stated

data-quality-loop (data)

The team standardized human evaluator criteria to improve ground truth quality, which then improved the automated evaluation loop.

“we ended up creating a standard operating procedure for the manual evaluators that would also be shared with the automated system and so we could have consistent results.”transcript, stated

human-in-the-loop-gate (safety)

The streaming system explicitly plans for safe failure modes that can trigger human involvement when the autonomous system hits new or uncertain situations.

“you do want to have safe ways that the system can fail and trigger the right sort of, you know, human involvement as needed.”transcript, stated

Tradeoffs

How broad each artwork moderation pass should be.

Chose one defect type at a time over multi-defect detection in a single large-context pass.

Rationale: Narrow passes made the defect problem tractable and avoided precision local maxima caused by mixed criteria.

“we broke the problem down into the individual defect types, ranked them by how often they occurred and how much effort it was to manually perform the detection and tackle them one at a time.”transcript, stated

How autonomous the streaming-quality agent should be.

Chose autonomous mitigation with guardrails and human fallbacks over unconditional autonomous action.

Rationale: Streaming failures need fast action, but the team still plans for failure modes and brand-new situations that require human evaluation.

“we want to have an autonomous system but there are going to be times when you might want to have sort of human evaluation”transcript, stated

How much operational context to give streaming-quality agents

Chose curated high-quality context over maximal logs and metrics regardless of relevance.

Rationale: The source says data quality beats quantity, so curation protects reasoning quality.

“data quality always beats data quantity when you're trying to develop such a system”transcript, stated

Gotchas (what bit them)

Inconsistent manual moderation criteria poisoned the evaluation data.

Root cause: Manual evaluators did not always follow the same standard operating procedure, so similar artwork could be labeled differently.

Resolution: Create a shared SOP for human evaluators and the automated system to produce a better ground truth dataset.

“the manual evaluators were using inconsistent criteria. Some of them would pass something that others would fail and vice versa.”transcript, stated

Trying to detect too many artwork defect types at once hurt performance.

Root cause: The team overused large context windows and mixed multiple defect-detection problems together.

Resolution: Break the problem into individual defect types, rank by frequency and manual effort, and tackle them one at a time.

“we initially tried to take advantage of that. We tried to run defect detection for multiple types of issues at once and found that that was just not the way to go.”transcript, stated

Adding more operational data can make an agentic incident system worse.

Root cause: Infrastructure logs, metrics, past incidents, and tickets compete for context-window budget.

Resolution: Prioritize high-quality context that directly helps reach the right operational outcome.

“data quality always beats data quantity when you're trying to develop such a system”transcript, stated

Evaluation

Prime Video reports evaluation-loop improvements for artwork moderation and lessons from streaming-quality agent development.

Metrics: 70% reduction in false positives and negatives · evaluation/results time from several days to less than an hour in some cases · 88% reduction in manual effort · 18 million concurrent viewers for a Thursday Night Football example

“we were also able to reduce the amount of time it takes to get results from several days to less than an hour for certain circumstances. But most importantly we were able to reduce that manual effort by 88%.”transcript, stated

Outcomes

AWS/Prime Video conference talk; reported metrics are internal and the two use cases have different maturity levels.

“So it kind of takes all of these logs which can then be used for sort of, you know, reflective analysis in terms of improving some of the decision-making.”transcript, stated