Scaling ML Annotation Platform with LLMs for Content Classification

spotify · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: content annotation platform

Spotify needed to generate high-quality annotations at catalog scale so ML teams could train and evaluate models for tracks, podcast episodes, content relations, and policy violations.

Constraints:

• Spotify models cover hundreds of millions of tracks and podcast episodes.
• Annotation workflows had to support both human domain expertise and configurable LLM automation.
• Different projects needed flexible interfaces, quality metrics, access control, and integration with ML workflows.

Architecture

Spotify built an annotation platform around three pillars: scaling human expertise, annotation tooling, and workflow/ML integration. Domain expert workforces provide first-pass review; configurable LLM systems run in parallel to grow annotations; custom interfaces support classification, audio/video, and NLP tasks; backend services manage projects, access, distribution, metrics, agreement, escalation, and integration with batch orchestration and model training/evaluation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Content catalog"] --> B["Annotation sampling pipeline"]
  B --> C["Annotation tooling"]
  C --> D["Human expert workforces"]
  C --> E["LLM annotation system"]
  C --> F["Project metrics"]
  D --> G["Quality analysts"]
  F --> G
  F --> H["ML training and evaluation workflows"]

Components

Data flow

ML teams sample model predictions or content items, route them through annotation tooling, combine human expert review with configurable LLM annotations, compute project and agreement metrics, escalate ambiguous cases to quality analysts, and feed high-confidence annotations back into training and evaluation workflows.

Why this architecture

Spotify combined human and machine annotation because human expertise alone would not scale, while technical automation alone would miss domain nuance and quality resolution.

Models used: LLM-based annotation system · content classification models · content relation detection models · policy violation models

Scale

• Spotify models cover hundreds of millions of tracks and podcast episodes.
  “several ML models that cover entire content catalogs — hundreds of millions of tracks and podcast episodes.”article, stated
• The initial workflow increased the annotation corpus by 10x.
  “We increased the corpus of annotations by 10 times”article, stated
• Annotator productivity improved by 3x.
  “did so with three times the improvement in annotator productivity.”article, stated
• The platform target was millions of annotations.
  “build a platform that would scale to millions of annotations.”article, stated

Alternatives considered (roads not taken)

Ad hoc annotation scripts only

Rejected because: Spotify needed a platform strategy for millions of annotations and multiple ML tasks.

One annotation tool for every use case

Rejected because: No single tool fits all Spotify annotation needs, so abstractions had to support multiple tools and interfaces.

Human expertise without LLM automation

Rejected because: LLM automation significantly grew the corpus at lower effort and cost.

Patterns

human-in-the-loop-gate (annotation)

Human experts and quality analysts review first-pass labels and ambiguous cases.

synthetic-data-generation (data)

A configurable LLM annotation system expands the label corpus alongside human annotators.

eval-driven-development (evaluation)

Project metrics and agreement measures guide annotation quality and model training readiness.

batch-offline-processing (mlops)

Production annotation integrations connect to internal batch orchestration and ML workflow infrastructure.

Tradeoffs

Annotation strategy

Chose human expertise plus LLM-based automation over only humans or only machine annotations.

Rationale: The combined approach scales volume while preserving domain expertise and quality escalation.

Tooling architecture

Chose generic abstractions with multiple interfaces over single application for every annotation task.

Rationale: Spotify's use cases and tools are too varied for one fixed annotation UI.

Gotchas (what bit them)

Annotation tasks varied beyond simple classification.

Root cause: Spotify needed audio/video segment annotation, NLP tasks, and other complex project-specific workflows.

Resolution: Build custom interfaces and generic data models/APIs that support multiple annotation tools.

Experts may disagree on nuanced content labels.

Root cause: Tasks such as detecting overlaid music in a podcast clip can legitimately produce different expert opinions.

Resolution: Compute agreement metrics and escalate unresolved cases to quality analysts.

Scaling annotation work requires more than more annotators.

Root cause: Human workforces need tooling, distribution, metrics, access control, and backend project management.

Resolution: Build platform tooling around work distribution and project operations.

Evaluation

Spotify evaluates annotation work through project completion, data volumes, annotations per annotator, agreement metrics, escalation rates, and downstream model training/evaluation integration.

Metrics: project completion rate · data volumes · annotations per annotator · agreement metric · annotation corpus size · annotator productivity

Outcomes

• Spotify increased its annotation corpus by 10x.
• Annotator productivity improved by 3x.
• The platform supports dozens of annotation projects in parallel.
• The platform reduced time to develop new ML models and iterate on existing systems.

The source is a Spotify engineering blog with concrete platform metrics and a high-level description of LLM automation.