Automated Synopsis Generation Pipeline with Human-in-the-Loop Quality Control

netflix · Media & Entertainment · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Streaming content metadata

Netflix needed to scale synopsis creation for a large catalog without replacing writers or exposing low-quality LLM drafts that would waste editor time.

Constraints:

• writers need context from scripts, captions, viewables, and other title assets
• draft synopses must satisfy Netflix voice, syntax, clarity, style, and factual accuracy standards
• bad drafts increase editing time instead of saving it
• creative workers remain in the workflow
• model choices and prompt assets need to evolve as better models appear

Architecture

Netflix built an assistive synopsis generation workflow in Metaflow. An asset store and writer-curated prompt library summarize context sources such as closed captions, viewables, audio descriptions, and scripts. A second prompt library generates draft synopses using examples informed by member data. Several LLM-as-judge models then score each synopsis against five writer-defined quality criteria before accepted drafts reach the writer platform for human editing.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  context-source-store[Context source store] --> metaflow-generation-flow[Metaflow generation flow]
  prompt-library[Prompt library] --> metaflow-generation-flow
  metaflow-generation-flow --> draft-synopsis-generator[Draft synopsis generator]
  draft-synopsis-generator --> llm-judge-ensemble[LLM judge ensemble]
  llm-judge-ensemble --> writer-platform-api[Writer platform API]

Components

Data flow

Context sources for each title are summarized with prompts and writer-curated assets, then a synopsis-specific prompt library generates drafts. The evaluation flow applies several LLM judges over precision, accuracy, tone, clarity, and writing style. Passing drafts are exposed to the API and writer platform; failed drafts are blocked before writers spend time editing them.

Why this architecture

Netflix framed the system as assistive technology for writers rather than a replacement system. Separating generation from LLM-judge quality control lets the team scale draft creation while reducing the chance that bad drafts reach writers. Metaflow gives the team a workflow substrate where context gathering, prompt libraries, model swaps, and evaluation can be iterated independently.

Models used: LLM draft generator · several LLM-as-judge models · Llama or OpenAI models tested in comparisons

Scale

• Netflix produces multiple synopses for each show.
  “multiple synopses for each show we can weave and we can mix all of the information that we have for each show and this is a lot of”transcript, stated
• The evaluation step uses five quality criteria.
  “uh models that we use so these are the five criteria and the five models that we have um in that step um they're each”transcript, stated
• The team ran time studies and described the savings as significant enough for production.
  “sh share more but we did run time studies and um the the skill that we're able to achieve with this is pretty significant maybe I I'll phras another”transcript, implied
• Netflix kept the writer room and hired more writers while using automation to produce more drafts.
  “they are doing something else we kept the room we kept every single writer we actually uh hired more but because they can do more um and we can provide more”transcript, stated

Alternatives considered (roads not taken)

Show writers raw ChatGPT-style outputs

Rejected because: The team believed writers would reject unfiltered generic outputs.

Expose every generated synopsis to writers

Rejected because: Bad strings would waste writer time and damage trust in the tool.

Remove human writers from the workflow

Rejected because: Netflix explicitly kept humans in the loop and positioned LLMs as draft assistance.

Patterns

human-in-the-loop-gate (workflow)

Writers edit accepted synopsis drafts, and the system is explicitly designed to keep human creative review in the path.

llm-as-judge (evaluation)

Several models act as judges over writer-defined quality criteria before drafts reach writers.

eval-driven-development (evaluation)

Generation quality is controlled by pass/fail gates over precision, accuracy, tone, clarity, and writing style.

prompt-versioning (prompting)

Netflix maintains prompt libraries curated with writers and dedicated to different synopsis string types.

Tradeoffs

Keep writers in the loop

Chose assistive draft generation plus human editing over fully automated synopsis publication.

Rationale: Netflix wanted more synopses without replacing creative judgment.

Use multiple LLM judges

Chose separate pass/fail checks for five criteria over one generic quality score.

Rationale: Different quality dimensions fail for different reasons, from hallucination to tone mismatch.

Use modular model experimentation

Chose swap candidate models and compare writer preference over lock the system to one model choice.

Rationale: The team expected model capabilities to change quickly.

Gotchas (what bit them)

Bad drafts can make writers slower, not faster.

Root cause: Low-quality strings create editing burden and erode trust.

Resolution: Netflix added an LLM-judge QC flow before drafts reach the API.

Insufficient or poor context produces bad synopsis batches.

Root cause: The generation model depends on source context quality for factual and stylistic output.

Resolution: The judge catches failed drafts before exposure.

Writer acceptance required more than factual correctness.

Root cause: Tone, AP style, clarity, and Netflix voice all affect whether a synopsis is usable.

Resolution: The evaluation flow uses separate models/criteria for each dimension.

Evaluation

Netflix evaluates drafts with several LLM judges against five writer-defined criteria and uses writer feedback/time studies to assess workflow impact.

Metrics: precision · accuracy · tone · clarity · writing style · writer time savings · writer preference in model comparisons

Outcomes

• The system reached production.
• Netflix reported significant time savings but did not disclose exact numbers.
• Writers start from drafts rather than from scratch.
• Netflix kept and hired writers while increasing synopsis throughput.

The source withholds exact throughput and time-savings figures, but states the savings were significant enough for production.