Teaching Refusal Behavior Through Automated Data Curation with LLM Judge Consensus

shopify · E-commerce · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Merchant-facing AI assistant refusal behavior

Shopify Sidekick's customer segmentation skill was trained on successful production queries, so it did not know when to refuse impossible requests and instead produced hallucinated or zero-result segment queries.

Constraints:

• The skill model translates merchant language into Shopify segmentation syntax.
• Training data from successful queries lacked refusal examples.
• Mislabeling valid merchant requests as refusals would harm common workflows.
• Manual annotation needed to scale beyond the seed dataset.
• The refusal taxonomy had to be mutually exclusive to avoid label conflict.

Architecture

Shopify built an automated data-curation pipeline for refusal training. A seed set of 600+ human-annotated refusal examples calibrated an ensemble of four frontier LLM judges, strict consensus and a mutually exclusive taxonomy reconciled production logs with refusal labels, and the curated dataset fine-tuned the segmentation skill.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  outer_planner["Outer planner"] --> customer_segmentation_skill["Customer segmentation skill"]
  production_query_logs["Production query logs"] --> llm_judge_ensemble["LLM judge ensemble"]
  human_refusal_seed_dataset["Human refusal seed dataset"] --> llm_judge_ensemble
  llm_judge_ensemble --> consensus_filter["Consensus filter"]
  consensus_filter --> conflict_resolver["Conflict resolver"]
  production_query_logs --> conflict_resolver
  conflict_resolver --> curated_refusal_training_data["Curated refusal training data"]
  curated_refusal_training_data --> customer_segmentation_skill

Components

Data flow

Production customer-segmentation queries and human refusal seed labels enter the curation engine. Four frontier LLM judges classify each query into mutually exclusive categories, strict consensus filters labels, conflicts with production logs are resolved, and the resulting training set fine-tunes the Sidekick segmentation skill.

Why this architecture

Shopify needed automated curation because successful production logs could not teach refusal, while naively merging refusal labels caused conflicts; consensus judging traded coverage for label quality.

Models used: Sidekick segmentation skill model · Four frontier LLM judges

Scale

• The seed dataset included about 600 standard queries and 602 refusal annotations.
  “Shopify partnered with Toloka to produce a balanced dataset of approximately 600 standard queries and 602 refusal annotations.”summary, stated
• Evaluation score improved from 0.619 to 0.798.
  “improved segmentation evaluation scores from 0.619 to 0.798 (28.9% relative gain)”summary, stated
• The final model achieved 86.3% refusal accuracy with 4.6% false positives.
  “achieved 86.3% refusal accuracy with only 4.6% false positives”summary, stated

Alternatives considered (roads not taken)

Train only on successful production queries

Rejected because: That corpus could not teach the skill to refuse impossible requests.

Naively merge the refusal seed set with production data

Rejected because: It created label conflicts and failed to deliver meaningful improvements.

Scale manual annotation directly

Rejected because: Shopify used frontier LLM judges instead of manual scaling, with the human set as calibration seed.

Patterns

eval-driven-development (data quality)

Frontier LLM judges and evaluation scores guided refusal-data curation and model improvement.

safety-policy-enforcement (refusal behavior)

The model learned when to refuse requests that the segmentation skill could not satisfy.

human-in-the-loop-gate (annotation)

Human refusal annotations seeded and calibrated the automated judge process.

Tradeoffs

Use strict LLM-judge consensus.

Chose High-precision consensus labels over Higher-coverage but noisier labels.

Rationale: False positive refusals would damage legitimate merchant workflows.

Use automated judge curation instead of scaling human labels directly.

Chose Four-frontier-LLM judge ensemble over Manual annotation at full scale.

Rationale: The human dataset became a seed for a reusable curation flywheel over production traffic.

Gotchas (what bit them)

Production success logs can erase refusal behavior.

Root cause: The corpus contains examples of solvable requests, not impossible ones.

Resolution: Curate explicit refusal examples and reconcile them with production labels.

Adding refusal data naively creates contradictory labels.

Root cause: Semantically similar queries can receive both success and refusal signals without reconciliation.

Resolution: Use a mutually exclusive taxonomy and strict judge consensus to resolve conflicts.

False positive refusals are more damaging than missed refusals for common merchant workflows.

Root cause: A legitimate merchant request mislabeled as impossible blocks useful behavior.

Resolution: Prefer strict consensus and accept lower coverage in exchange for high precision.

Evaluation

Shopify measured segmentation evaluation score, refusal accuracy, false-positive rate, and label consistency after curation.

Metrics: 0.619 to 0.798 segmentation evaluation score. · 28.9% relative gain. · 86.3% refusal accuracy. · 4.6% false-positive rate. · 600+ human-annotated refusal seed examples.

Outcomes

• Segmentation evaluation improved from 0.619 to 0.798.
• Refusal accuracy reached 86.3%.
• False positives were limited to 4.6%.
• A continuous data flywheel was established for future training signal.

The case reports 600+ human refusal examples, four frontier LLM judges, and production traffic as ongoing training signal.