LLM-Powered Data Labeling Quality Assurance System

uber · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data-labeling quality assurance

Uber AI Solutions needed to catch labeling-quality defects during annotation instead of discovering them after completion, where rework increased cost, timelines, and customer friction.

Constraints:

• Client SOP documents contain many requirements that labelers must follow.
• Single broad LLM validation calls caused hallucinations and missed enforcements.
• Validation had to preserve client data privacy and avoid retaining SOP data.
• The system had to run inside annotation workflows without making labelers wait too long.

Architecture

Uber built Requirement Adherence as a two-phase LLM quality system. The first phase converts SOP documents into markdown and extracts atomic, self-contained rules with reflective LLM calls plus optional manual rules. The second phase runs in-tool validation per rule, routes deterministic and subjective checks to different model classes, uses prefix caching and parallel calls for latency, and keeps LLM calls stateless and privacy-preserving.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  SOP_converter["SOP markdown converter"] --> Rule_LLM["Rule extraction LLM"]
  Rule_LLM --> Manual_review["Manual rule review"]
  Manual_review --> Rule_validator["Rule-format validator"]
  Rule_LLM --> Rule_validator
  Rule_validator --> Validation_engine["In-tool validation engine"]
  Validation_engine --> Model_routing["Model routing"]
  Model_routing --> Annotation_workflow["uLabel annotation workflow"]

Components

Data flow

A client SOP is converted into markdown, decomposed into atomic rules by an LLM with reflection, optionally supplemented by manual rules, and normalized by another LLM call. During labeling, each rule is checked independently in parallel; code handles formatting checks, non-reasoning LLMs handle deterministic checks, and reasoning LLMs handle subjective checks.

Why this architecture

Uber decomposed the problem because post-labeling audits were expensive and monolithic LLM checks were unreliable. Atomic rule extraction plus per-rule validation lets the system choose the cheapest sufficient computation for each rule while catching defects inside the labeler's workflow.

Models used: LLMs with reflection · non-reasoning LLM · reasoning LLM

Scale

• The system reduced required audits by 80%.
  “we observed a substantial enhancement in our in-tool annotation quality, which resulted in an 80% reduction in audits required”article, stated
• The validation step is standard across Uber AI Solutions' annotation pipeline and client base.
  “This validation is now a standard and widely adopted step within our annotation pipeline, used across our entire client base.”article, stated
• Prefix caching is used to reduce validation latency for labelers.
  “We rely on prefix caching to reduce the latency of the checks and create a better user experience.”article, stated

Alternatives considered (roads not taken)

Post-labeling checks and interhuman agreement only

Rejected because: They ensure quality after the fact but create rework, time loss, cost, and customer friction.

One monolithic LLM call for all SOP checks

Rejected because: The team observed hallucinations and missed enforcements when all requirements were checked together.

Use the same model for every rule type

Rejected because: Different rules need different computational power; formatting can be code, deterministic checks can use non-reasoning LLMs, and subjective checks need reasoning models.

Patterns

prompt-chaining (orchestration)

Requirement Adherence chains SOP conversion, rule extraction, reflection, manual-rule validation, and per-rule runtime checks instead of using one broad prompt.

human-in-the-loop-gate (safety)

Humans can add rules missed by the LLM or absent from SOPs before the validation rule set is finalized.

model-routing (cost-performance)

The system routes deterministic, subjective, and complex subjective checks to cheaper or more capable model paths according to rule complexity.

privacy-preserving-processing (safety)

LLM calls are stateless and do not retain client SOP data for training.

Tradeoffs

Validate per rule rather than all rules at once

Chose single validation call per rule over single monolithic LLM call.

Rationale: Per-rule checks reduce missed enforcements and can run in parallel to keep labeler feedback fast.

Route checks by rule complexity

Chose code, non-reasoning LLMs, and reasoning LLMs for different checks over one expensive reasoning model for all validation.

Rationale: The team wanted to maximize both accuracy and cost-efficiency by applying only the needed computation.

Gotchas (what bit them)

A single LLM validation prompt missed requirements and hallucinated.

Root cause: Checking all SOP requirements at once overloaded the model with too many constraints.

Resolution: Extract atomic rules and run one validation call per rule.

Post-labeling quality checks created costly rework.

Root cause: Errors were caught after labeling had already been completed.

Resolution: Move validation into the labeling tool and give suggestions during annotation.

Auto-extracted rule sets could miss requirements or overlap with human additions.

Root cause: SOP documents and client requirements may contain gaps, ambiguity, or manually supplied constraints.

Resolution: Allow manual rules, then validate their format and overlap with another LLM call.

Evaluation

Uber evaluated the system through audit reduction, annotation-quality improvement, latency optimization, and adoption as a standard validation step across the annotation pipeline.

Metrics: 80% reduction in audits required · validation standard across the entire client base · prefix caching for latency reduction

Outcomes

• Requirement Adherence reduced audit needs by 80%.
• In-tool validation became a standard step across the annotation pipeline.
• The system improved timelines and costs while preserving client data privacy.

The article frames the system as deployed across Uber AI Solutions' enterprise labeling client base.