Challenges in Designing Human-in-the-Loop Systems for LLMs in Production

v7 · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Human-in-the-loop learning systems for production LLM and multimodal workflows

V7 argues that production human-in-the-loop systems for LLMs are still primitive: most feedback is thumbs-up/down, automation is overrated, expert users are hard to teach from, and many industrial tasks require discrete outcomes with little tolerance for model error.

Constraints:

• V7 works with ground-truth data for many AI companies, so the talk is grounded in training-data practice rather than a single deployment metric.
• Many industrial tasks have no easy undo path, making broad automation risky.
• Human experts may provide feedback that is incorrect, incomplete, or badly shaped for model training.
• Information asymmetry can make an answer wrong for one user even when it is not intrinsically false.

Architecture

The practical system described is a human-in-the-loop labeling platform with Auto Label as a multimodal co-pilot: a human provides language instructions, a large model or vision model proposes labels or segmentations, the workspace shows visual output, and expert review/feedback remains necessary because automation, teaching quality, and information targeting are unsolved production problems.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  expert-labeler["expert labeler"] --> labeling-workspace["labeling workspace"]
  labeling-workspace --> auto-label-copilot["Auto Label copilot"]
  auto-label-copilot --> vision-or-multimodal-models["vision or multimodal models"]
  vision-or-multimodal-models --> training-data-store["training data store"]
  training-data-store --> labeling-workspace
  expert-labeler --> feedback-channel["feedback channel"]
  feedback-channel --> auto-label-copilot
  expert-labeler --> production-decision-surface["production decision surface"]

Components

Data flow

An expert labeler gives an instruction inside V7's labeling workspace; Auto Label interprets the prompt and image context, applies a suitable model to segment or label visual objects, returns visual labels for review, and the human correction/QA path becomes training or feedback data rather than a simple thumbs-up/down signal.

Why this architecture

V7 keeps humans in the loop because the talk argues that LLMs are still often used as glorified zero-shot models, industrial tasks need discrete reliable outcomes, and experts remain necessary when automation cannot safely handle labeling decisions.

Models used: LLMs · multimodal models · classical machine-learning models · fine-tuned deep-learning models

Scale

• V7 handles ground truth for hundreds of AI companies.
  “we handle the ground truth of hundreds of AI companies”transcript, stated
• V7 says it has petabytes of well-labeled training data.
  “We're talking about petabytes of training data that is also really well labeled”transcript, stated
• The talk frames current production feedback as often limited to thumbs up/down despite important information.
  “sometimes deployed in production and sometimes handling really important information.”transcript, stated

Alternatives considered (roads not taken)

Simple thumbs-up/down feedback

Rejected because: The speaker says this leaves a huge amount of untapped potential for improving systems that may handle important production information.

End-to-end automation or QA-only co-pilots

Rejected because: Automation takes a long time to reach production, and LLMs make it harder because their impressive language can mask limited reliability.

LLM interface for every workflow

Rejected because: Some software already has precise action interfaces, and a classical/fine-tuned model may be enough for the actual discrete action.

Patterns

human-in-the-loop-gate (control)

The system relies on expert labelers to review, correct, and teach models because full automation is unreliable in high-stakes labeling contexts.

multi-agent-orchestration (coordination)

The talk frames co-pilots as assistants to human professionals rather than autonomous replacements, with the human and model sharing task execution.

fine-tuning-adaptation (model-adaptation)

The speaker repeatedly contrasts large models with smaller models fine-tuned on bespoke data for industry tasks.

data-quality-loop (training-data)

The core product concern is how human feedback and ground-truth labels become better data for model learning over time.

Tradeoffs

Use Auto Label as a co-pilot rather than full replacement.

Chose Human-reviewed multimodal co-pilot labeling over Fully automated expert labeling.

Rationale: If the expert work is easy enough to automate fully, the speaker argues the user likely should not be labeling that data in the first place.

Prefer smaller fine-tuned/classical models for some discrete tasks.

Chose Task-specific models where the outcome is simple and well-defined over Using a large model interface for every action.

Rationale: Many industrial workflows only need a discrete action, so the broad reasoning surface of an LLM can be unnecessary.

Keep humans involved despite high automation ambition.

Chose Human-in-the-loop product experiences over Immediate end-to-end automation.

Rationale: The talk says automation takes a long time to reach production and large models still fail on reliability, teaching, and context targeting.

Gotchas (what bit them)

Thumbs-up/down feedback is too weak for production learning loops.

Root cause: Binary critique does not capture the rich correction, context, and teaching signal needed to improve systems.

Resolution: Design richer human-computer interaction around labeling, teaching, and information transfer.

Automation can be overrated when production decisions have no undo.

Root cause: Industrial tasks often involve physical or financial consequences with strictly defined outcomes.

Resolution: Keep automation bounded and use human review where mistakes cannot be safely reversed.

Humans are not automatically good model teachers.

Root cause: User-provided retraining feedback may be incorrect or poorly formatted for how RLHF/model learning works.

Resolution: Treat human teaching as a product-design problem, not a raw feedback channel.

Correct information can still be wrong for the user.

Root cause: Information asymmetry means the same answer may be inappropriate depending on who receives it.

Resolution: Account for user role, expertise, and context in human-in-the-loop design.

Evaluation

The talk evaluates production HITL maturity through observed industry practice, V7's ground-truth platform experience, and qualitative failure modes rather than a formal benchmark.

Metrics: hundreds of AI companies' ground-truth workflows · petabytes of labeled training data · presence of production feedback mechanisms · automation reaching production · discrete outcome correctness · quality of human teaching signal

Outcomes

• V7's Auto Label demonstrates a multimodal co-pilot workflow for visual labeling.
• The talk concludes that production HITL systems remain early and often over-rely on simplistic feedback.
• The most reusable outcome is a set of design cautions around automation limits, poor teaching signals, information asymmetry, and discrete production outcomes.

V7 reports handling ground truth for hundreds of AI companies and petabytes of well-labeled training data; the talk does not provide Auto Label adoption or accuracy metrics.