Human-AI Co-Annotation System for Efficient Data Labeling

appen · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: data annotation and human-AI labeling workflows

Appen needed a scalable way to improve annotation throughput and cost-effectiveness for complex generative AI data projects without sacrificing quality, especially as data management, labeling, and lack of domain-specific data became larger bottlenecks.

Constraints:

• generative AI projects require larger volumes, more diverse data sources, and more complex preparation than traditional ML
• customized LLM applications often require labeling unstructured documents, PDFs, charts, diagrams, and context-heavy data
• LLMs are useful for well-structured repetitive labeling but struggle with ambiguity, nuance, cultural context, domain expertise, and multi-step reasoning
• recursive training on model-generated data creates quality risks
• human annotation remains the quality gold standard but is slower and more expensive
• quality thresholds must route uncertain model predictions to humans while still auditing confident model predictions
• co-annotation should reduce cost and time while maintaining an acceptable accuracy level

Architecture

Appen described a human-AI co-annotation strategy and platform workflow. A dataset, ground-truth/few-shot examples, business objectives, and annotation criteria feed a co-annotation engine that can use a chosen LLM or ML resource for pre-labeling. Model uncertainty, measured with entropy across prompt variations or model predictions, routes predictions: low-entropy/high-confidence labels can go directly to output, high-entropy labels go to human review, and a subsample of low-entropy labels is still checked by humans for quality. Finalized annotations become new ground truth or few-shot examples to improve later model performance. Appen's Model Mate feature lets users attach one or multiple LLMs at steps inside ADAP annotation tasks, for example generating prompt ideas, screening prompts, producing pre-annotations, helping science/math formatting, generating red-team topics, or diff-checking code.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Dataset to label"] --> D["Co-annotation engine"]
  B["Ground truth and few-shot examples"] --> D
  C["Business objectives and annotation criteria"] --> D
  D --> E["LLM pre-annotation model"]
  E --> F["Uncertainty calculator"]
  F --> G["Quality threshold router"]
  G --> H["Human annotators"]
  G --> K["Finalized annotation data"]
  H --> K
  K --> B
  I["ADAP platform"] --> J["Model Mate integration"]
  J --> E
  I --> H

Components

Data flow

The workflow begins with a dataset, ground-truth labels used as few-shot examples, business goals, and annotation criteria. The co-annotation engine uses a selected model to generate label predictions. An uncertainty calculation compares prediction variability across prompts or model runs; low-entropy predictions pass the quality threshold, high-entropy predictions route to human annotators, and a human sample audits confident labels. Human-reviewed and accepted labels are finalized, then reused as ground truth or few-shot examples. In the case study, Model Mate sent search queries, product titles, and product images to GPT-4o at page load, returned real-time label suggestions and explanations to experienced search-relevance contributors, and measured the resulting accuracy, time, and cost changes.

Why this architecture

Appen chose uncertainty-routed co-annotation because LLMs can cheaply pre-label easy, repetitive, well-structured items, but humans are still needed for ambiguous, nuanced, domain-specific, or high-risk examples. Entropy gives a routing signal for when model output is likely reliable, while human review and subsampling protect against over-trusting confident model predictions. Model Mate embeds this pattern directly into annotation tasks so contributors receive suggestions or checks where they are useful rather than replacing the whole workflow.

Models used: GPT-3.5 Turbo · GPT-4o · multimodal LLM · LLMs · Model Mate-connected models

Scale

• Appen says its contributor crowd includes more than one million contributors worldwide.
  “you have a global crowd of more than one million contributors worldwide”transcript, stated
• The contributor base spans 235 languages.
  “different languages I think 235 total”transcript, stated
• Appen's survey reported a 177% increase in generative AI adoption in 2024.
  “in 2024 we saw 177% increase in the adoption of generative AI”transcript, stated
• The survey reported project deployment rates dropping by over 8% since 2021.
  “the percentage of projects making it to deployment dropped by over 8% since we started the survey in 2021”transcript, stated
• The example co-annotation calculation uses 1,000 utterance rows.
  “we have a thousand rows of utterances and llm it cost it less than a cent per row while human anotation is 45 cents per row”transcript, stated
• The example threshold achieved 87% accuracy versus 95% full-human annotation.
  “at you know a lower uncertainty threshold you know less than 30% we achieved an 87% accuracy”transcript, stated

  “which is competitive um with a full human adaptation at 95%”transcript, stated
• The example reduced human labor time from 150 hours to 55 hours, a 63% decrease.
  “we go from 150 hours of human hours to 55 hours combined right that's a 63% decrease in time and effort”transcript, stated
• The example reduced cost from $450 to $169, a 62% cost reduction.
  “reduce the costs from $450 to $169 which is 62% cost reduction”transcript, stated
• The electronics-company case targeted 95% balanced accuracy.
  “the customer wanted to achieve 95% balanced accuracy across different components within their task”transcript, stated

Alternatives considered (roads not taken)

Use LLM labels for all units without review

Rejected because: LLMs can fail on ambiguity, nuance, domain expertise, reasoning, and recursive data risks, so uncertain or sampled outputs need human review.

Send every unit to humans

Rejected because: Full human annotation preserves quality but has much higher cost and time than an uncertainty-routed hybrid workflow.

Rely on model self-reported confidence

Rejected because: The speaker warns that self-reported confidence scores are not consistently good predictors of annotation quality.

Use a fixed threshold without business-risk tuning

Rejected because: The uncertainty threshold directly controls cost, human review volume, and accuracy, so teams need to choose based on acceptable risk.

Patterns

human-in-the-loop-gate (workflow)

High-uncertainty predictions are routed to human annotators, multiple human judgments can be used to reach agreement, and even low-entropy outputs receive human subsampling for consistency.

confidence-thresholding (routing)

Entropy or uncertainty thresholds decide whether an LLM prediction passes directly, gets routed to human review, or needs sampled human verification.

data-quality-loop (data improvement)

Finalized human/AI labels become new ground truth or few-shot examples, improving later model performance and aligning annotation data with business criteria.

cost-aware-routing (operations)

The co-annotation router uses uncertainty thresholds to trade off accuracy, cost, and time by sending only harder items to human annotators.

eval-driven-development (evaluation)

Appen measured accuracy against SME-labeled ground truth, tested prompt variations, ran A/B evaluation for model-assisted versus unassisted contributors, and reported accuracy/cost/time impacts.

synthetic-data-generation (augmentation)

Model Mate can generate prompt ideas, harmful red-team topics, and pre-annotations inside annotation tasks, using AI to create or prepare data for humans to refine.

Tradeoffs

Route by uncertainty threshold

Chose low-entropy model labels pass, high-entropy labels go to human review over all-human annotation or all-LLM annotation.

Rationale: The threshold lets teams tune the balance between accuracy, human effort, time, and cost.

Use entropy from prompt variation

Chose prediction variability across prompt variations over single deterministic label from one prompt.

Rationale: Different prompts reveal whether the model consistently maps a unit to the same category, which becomes the uncertainty signal for routing.

Use AI suggestions inside the annotation task

Chose Model Mate pre-annotations, prompt inspiration, screeners, and validators over keeping model assistance outside the contributor workflow.

Rationale: Embedding the model directly in ADAP lets contributors use model output as reference while still making their own decisions.

Use experienced contributors for the electronics search relevance case

Chose experienced search-relevance contributors plus GPT-4o multimodal suggestions over generic crowd-only or model-only labeling.

Rationale: The client targeted 95% balanced accuracy, so Appen combined expert human review with multimodal model assistance and streamlined guidelines.

Gotchas (what bit them)

LLM confidence reported by the model may not predict annotation quality.

Root cause: Self-evaluation and self-reported confidence can be poorly calibrated for labeling correctness.

Resolution: Appen recommends entropy or uncertainty across predictions plus human review, rather than relying only on self-reported confidence.

LLMs are not universally good annotators.

Root cause: They struggle with ambiguous language, nuance, context, cultural references, domain expertise, and multi-step reasoning.

Resolution: Route difficult examples to human annotators and use AI where it is strongest: pre-labeling, suggestions, repetitive labels, and screening.

Model-generated labels can bias humans if not checked.

Root cause: A central concern is that incorrect LLM labels might degrade human accuracy during assisted annotation.

Resolution: Appen A/B tested this and found in the cited case that human accuracy remained unaffected when the LLM labels were incorrect.

Generative AI labeling demand outgrew older data-prep assumptions.

Root cause: Customized LLMs require diverse unstructured data, domain context, and more complex collection and annotation than traditional structured ML data.

Resolution: Appen positions co-annotation and Model Mate as a way to combine AI assistance, workflow routing, SMEs, dashboards, validation, and human review.

Evaluation

Appen evaluated co-annotation with a threshold/cost model on a 1,000-utterance example, ground-truth comparisons against SME-labeled test data, entropy across prompt variations, and an A/B evaluation in a search-relevance case study for a leading US electronics company using GPT-4o multimodal pre-annotations.

Metrics: 87% accuracy at less than 30% uncertainty threshold · 95% full-human annotation accuracy comparator · LLM cost less than one cent per row · human annotation cost 45 cents per row · 8 seconds per LLM annotation versus 180 seconds per human annotation · 150 human hours reduced to 55 combined hours · 63% decrease in time and effort · $450 reduced to $169 · 62% cost reduction · 3-4 percentage point accuracy increase across task components · accuracy rose to 94% from 90% without assistance

Outcomes

• In the example cost/threshold exercise, hybrid annotation reduced effort by 63% and cost by 62% while retaining 87% accuracy versus 95% full-human accuracy.
• In the electronics search relevance case, LLM-assisted contributors improved accuracy by 3-4 percentage points across three task components.
• Accuracy reached 94% with LLM labels and explanations, compared with 90% without assistance.
• When LLM labels were incorrect, human accuracy remained unaffected in the reported A/B evaluation.
• Model Mate enabled LLM use inside ADAP tasks for pre-annotation, prompt generation, screening, red-team topic generation, and code diff checking.

The approach is positioned for Appen's ADAP platform, global crowd of more than one million contributors across about 235 languages, and customer labeling projects involving text, image, audio, video, geospatial, search relevance, math/science, red teaming, and code review tasks.