LLM as Judge

Definition

LLM as judge uses a model to apply an explicit rubric or checklist to another model's behavior, conversation, or artifact. The judge produces an evaluation signal rather than the task result itself. Sierra's evaluator agent checks simulated conversations against checklists, while Sword supplies the judge with the task, transcript, rubric item, and evaluation instructions. Sierra Sword Health

Corpus Support

The analysis counts 120 deduplicated agentic systems (12.1%) across 13 industries. Evidence includes 69 article, 93 transcript, and 18 summary quotes; 179 are stated and 1 implied. Corpus analysis

Problem It Solves

Many agent qualities are too semantic or subjective for exact assertions, yet human review does not scale to every run. Shopify builds specialized judges for separate performance dimensions and calibrates them against human labels; Parlance uses an oracle model for synthetic evaluation only after measuring agreement with people. Shopify Parlance Labs

Applicability Signals

• Success is expressible as a rubric but not a deterministic assertion. Sword Health
• Conversation or workflow quality requires semantic interpretation. Sierra
• Human-labeled examples exist for calibration. Shopify
• Review volume exceeds expert capacity but important cases can still be audited. Parlance Labs

Architecture and Variants

• Checklist judge: evaluate a simulated conversation against explicit requirements. Sierra
• Rubric-specific judges: use separate evaluators for distinct quality dimensions. Shopify
• Oracle-model evaluation: compare synthetic model judgments with human evaluation before scaling them. Parlance Labs
• Bounded reflection: let a judge inspect output under business rules and checkpoints without granting it final authority. Fitch

Why Teams Choose It

Judges turn qualitative criteria into repeatable signals that can run over simulations and production traces. They can increase coverage between expensive human reviews, provided their agreement and failure modes are measured rather than assumed. Shopify Sword Health

Tradeoffs

• Judge coverage scales cheaply, but a judge can reproduce model bias or reward superficial compliance. Fitch
• Specialized judges improve criterion fit but multiply prompts, calibration sets, and maintenance. Shopify
• Binary criteria simplify gating while losing nuance in genuinely graded outcomes. Sword Health

Gotchas and Resolutions

• An uncalibrated judge can be barely better than random; Shopify measured and iterated from low agreement toward human levels. Shopify
• A model judging model output can be self-consistently wrong; retain human review and business-rule checkpoints. Fitch
• One universal evaluator misses criterion-specific failure modes; choose deterministic, heuristic, model, or human evaluators per criterion. Sword Health

When Not to Use It

Do not use a judge where a deterministic validator can directly test the requirement, or grant it final authority for high-stakes financial, clinical, or policy decisions. Fitch explicitly bounds reflection with business rules rather than delegating all decision-making to the evaluator model. Fitch

Boundaries and Related Patterns

llm-as-judge is one evaluator implementation. eval-driven-development is the broader practice of datasets, criteria, release gates, production monitoring, and feedback loops. Human expert review remains the calibration and escalation path rather than another judge implementation. Shopify Sword Health

Operational Checklist

• Define one explicit rubric and output contract per judge. Sword Health
• Measure agreement against representative human labels. Shopify
• Audit false positives, false negatives, and reward-hacking examples. Shopify
• Keep deterministic checks and accountable humans for hard boundaries. Fitch

Representative Implementations

Methodology and Limitations

This page samples 120 deduplicated systems. Public reports emphasize judge designs more than longitudinal drift and disagreement rates, so the pattern does not imply that model evaluation is interchangeable with expert judgment. Corpus methodology