Designing Human-AI Interaction for Critical Decision-Making in High-Stakes Testing

duolingo · Education · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: High-stakes online testing and human-AI review

Duolingo English Test needed human proctors to evaluate AI cheating signals independently, but trained reviewers were rubber-stamping false AI flags in a high-stakes exam context.

Constraints:

• Test results influence college admissions and visa decisions.
• The exam is fully online and remotely proctored.
• AI monitoring predicts possible cheating, but human proctors make final review decisions.
• Automation bias can corrupt both immediate decisions and downstream model/evaluation data.
• The system needed improvement without simply changing the model or adding more humans.

Architecture

Duolingo treated the human-AI interaction layer as the system component to optimize. The team inserted false AI cheating signals into historical sessions, measured how proctors responded, and found automation bias despite high reviewer calibration scores and a low model false-positive rate. They then changed proctoring guidelines so AI signals were framed as preliminary alerts and proctors had to find independent video evidence before upholding a flag. The result was a 21% increase in false-flag rejection rates.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  test-session-video["Test session video"] --> ai-cheating-signal["AI cheating signal"]
  ai-cheating-signal --> human-proctor-review["Human proctor review"]
  ai-cheating-signal --> interaction-guidelines["Interaction guidelines"]
  interaction-guidelines --> human-proctor-review["Human proctor review"]
  human-proctor-review --> final-decision["Final decision"]
  final-decision --> evaluation-and-training-data["Evaluation and training data"]
  evaluation-and-training-data --> ai-cheating-signal

Components

Data flow

The DET pipeline combines online exam video, AI-assisted cheating signals, and human proctor review. AI outputs enter a human interaction loop; proctor decisions become analytics, labels, training data, and future model/evaluation signals. By changing the interaction copy, Duolingo altered reviewer behavior and improved the quality of both decisions and downstream data.

Why this architecture

The team did not focus on model accuracy because the model already had a low false-positive rate; instead, it targeted the interaction loop where human behavior converted AI alerts into decisions and data. The copy change forced deliberation and independent video evidence, reducing rubber-stamping without changing the model or UI architecture.

Models used: AI-assisted cheating detection model (unspecified) · Duolingo English Test proctoring workflow

Scale

• The Duolingo English Test is trusted by 6,000 programs worldwide.
  “it still provides high-quality results that 6,000 programs worldwide are trusting day-to-day.”transcript, stated
• Human reviewers scored above 90% on accuracy calibration metrics.
  “our human reviewers are consistently scoring above 90% on their accuracy calibration metrics”transcript, stated
• The inserted false signals were accepted 50% of the time before the guideline change.
  “they actually accepted 50% of these fake signals, meaning they were falsely accusing people of cheating half half the time.”transcript, stated
• The model had a 1% false positive rate.
  “Our model has a 1% false positive rate”transcript, stated
• The copy change produced a 21% increase and raised correct rejection to 71%.
  “This simple copy change led a 21% increase in rejection rates, meaning that from 50% only half of sessions that were rejected uh the majority 71 were now determined to be totally okay.”transcript, stated

Alternatives considered (roads not taken)

Improve the model first

Rejected because: The team concluded the model was not the main problem because it had a 1% false positive rate.

Blame reviewer skill

Rejected because: The proctors were highly skilled and scored above 90% on calibration; the issue was interaction design.

Treat human-in-the-loop as a linear model-to-human-to-decision path

Rejected because: Duolingo frames the process as cyclical because human behavior creates eval/training data that affects future models.

Patterns

human-in-the-loop-gate (workflow)

Human proctors were final decision-makers, but the interaction had to require independent evidence rather than blind AI-signal acceptance.

eval-driven-development (evaluation)

The team ran an experiment with deliberately inserted false signals and measured reviewer behavior to find the system bottleneck.

data-quality-loop (data)

Human decisions are treated as data that feeds analytics, model improvements, and future evals, so interaction quality protects label quality.

guardrails-io-filtering (interaction-design)

Guidelines framed model outputs as preliminary alerts and required independent evidence before a flag could be upheld.

Tradeoffs

Add friction to proctor review

Chose Structured guidelines and review gates over Fast rubber-stamp acceptance of AI flags.

Rationale: In a high-stakes exam, slower deliberation is acceptable if it prevents false cheating accusations.

Patch guidelines instead of the model

Chose Interaction-copy redesign over Model tuning or UI overhaul.

Rationale: The failure was in how humans interpreted AI signals, and a small copy change produced a measurable behavior shift.

Treat human corrections as a measurement problem

Chose Capture human edits/diffs and questions over Only logging final binary decisions.

Rationale: The useful signal for future models is where humans disagreed, edited, or needed explanations.

Gotchas (what bit them)

High reviewer calibration did not prevent automation bias.

Root cause: The interaction design encouraged proctors to defer to AI signals instead of seeking corroborating evidence.

Resolution: Change the guidelines to slow reviewers down and require independent video evidence.

Human-in-the-loop systems can pollute their own data.

Root cause: If the system records only the final yes/no decision, it can miss edits, disagreements, and false signals.

Resolution: Capture the diff between AI suggestion and human correction so labels reflect what the human actually changed.

A correct perception signal is not necessarily a valid policy violation.

Root cause: The model may correctly detect headphones/earbuds-like pixels, but the object could be a permissible hearing aid.

Resolution: Split the interaction into whether the model signal is true and whether the human should apply the violation flag.

Evaluation

Duolingo inserted false AI signals into historical sessions, measured proctor acceptance/rejection, compared results to model false-positive and reviewer calibration metrics, and then measured the effect of guideline copy changes.

Metrics: Reviewer calibration accuracy above 90% · Model false positive rate of 1% · False-signal acceptance/rejection rate · 21% increase in rejection rates after copy change · 71% correct rejection after guideline update

Outcomes

• Identified automation bias as the real bottleneck rather than model accuracy or reviewer skill.
• Raised false-flag rejection from 50% to 71% through a guideline copy change.
• Established that human-AI interaction design controls both decision quality and downstream model/eval data quality.

The Duolingo English Test is trusted by 6,000 programs worldwide, and the studied human-AI loop affects high-stakes admissions and visa decisions.