Scaling AI Evaluation for Legal AI Systems Through Multi-Modal Assessment

harvey · Legal · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: legal AI evaluation for high-stakes professional work

Harvey needed to evaluate legal AI systems that retrieve sources, generate answers, analyze documents, and cite law in contexts where errors can undermine professional trust.

Constraints:

Architecture

Harvey built a three-pillar evaluation system: direct legal-expert review, automated evaluation pipelines, and a dedicated evaluation-data service. Experts create golden retrieval datasets and judge answer quality through anonymized A/B and Likert comparisons; automated evaluators use domain documentation, rubrics, confidence scores, canary runs, production monitoring, and citation-matching pipelines; the data service isolates evaluation data, standardizes annotations, enforces row-level access controls, and locks published datasets for reproducible experiments.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Legal experts"] --> B["Golden retrieval datasets"]
  B --> C["Retrieval systems"]
  C --> D["Answer generation system"]
  D --> E["Side-by-side evaluation tool"]
  E --> A
  D --> F["Automated evaluation pipelines"]
  F --> G["Knowledge source identification"]
  A --> H["Evaluation data service"]
  F --> H
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Legal experts Partners, tax experts, and subject-matter specialists who create datasets and judge outputs. Golden retrieval datasets, Side-by-side evaluation tool, Evaluation data service
Golden retrieval datasets Expert-curated query sets and relevant supporting documents for search and retrieval testing. Retrieval systems, Automated evaluation pipelines
Retrieval systems Traditional and agent-based search systems evaluated with precision, recall, NDCG, retrieval power, and model context. Answer generation system, Golden retrieval datasets
Answer generation system Harvey AI answer pipeline that retrieves documents, constructs context, and generates legal responses. Retrieval systems, Side-by-side evaluation tool, Automated evaluation pipelines
Side-by-side evaluation tool Human evaluation interface for anonymized A/B preference tests and Likert ratings. Legal experts, Answer generation system
Automated evaluation pipelines Nightly canaries, production monitoring, model vetting, rubric-based grading, and confidence scoring. Answer generation system, Knowledge source identification
Knowledge source identification Citation-verification subsystem using structured parsing, retrieval, embeddings, and LLM binary matching. Automated evaluation pipelines
Evaluation data service Isolated platform for data storage, access control, annotation, versioning, and collaboration. Legal experts, Automated evaluation pipelines

Data flow

Legal experts define queries and relevant documents for retrieval evaluation, then review generated answers through structured comparison tools. Automated systems grade model outputs using the user request, model answer, domain knowledge, expert prior knowledge, and rubrics. Citation strings are parsed into structured metadata, retrieved through databases or embeddings, and checked by an LLM binary matcher. Evaluation samples and annotations flow through a dedicated data service with versioning and role-based access control.

3 supporting quotes
“These evaluations follow a consistent two-step process that mirrors our answer pipeline—the end-to-end system that retrieves documents, constructs context, and generates answers.”article, stated
“Our automated evaluation systems consider several key elements: the model’s output, the original user request, relevant domain documentation or knowledge bases, and expert-provided prior knowledge.”article, stated
“The process starts with structured metadata extraction from each citation, parsing details such as the title, source collection, volume/issue (if any), page range, author/organization, and publication date.”article, stated

Why this architecture

Harvey combines expert review with automation because legal evaluation needs both professional judgment and broad continuous coverage. Expert evaluation supplies legal correctness and statistical release signals, while automated pipelines catch regressions, run nightly canaries, vet new models, and verify citations at scale. The separate data service protects confidentiality and keeps experiments reproducible.

3 supporting quotes
“By combining expert-led reviews with automated evaluations, we close critical gaps that either approach might leave on its own—ensuring broad evaluation coverage without sacrificing depth or rigor.”article, stated
“A recurring challenge in complex, multi-stakeholder evaluation processes is tracking where different datasets reside and controlling who can access them.”article, stated
“once a collection of evaluation samples is “published,” it becomes locked, ensuring that iterative experiments have a fixed baseline for comparing new features, models, or methods.”article, stated

Models used: GPT-4.1 · GPT-4o · LLM-based grading systems · LLM binary document-matching evaluation · embedding retrieval

Scale

Alternatives considered (roads not taken)

Expert review alone

Rejected because: Manual expert review cannot cover the volume of potential test cases or provide continuous monitoring.

“Data Scarcity: The sheer volume of potential test cases surpasses what any single expert or team can reasonably evaluate.”article, stated

Generic automated benchmarks

Rejected because: Legal workflows require nuanced rubrics and domain knowledge beyond generic evaluation.

“These systems go beyond generic benchmarks to capture the nuanced demands of professional legal workflows.”article, stated

Unstructured side-by-side review

Rejected because: Harvey uses anonymization, randomization, standardized prompts, and scales to reduce bias and measure statistical significance.

“These controls—randomized ordering, standardized prompts, and anonymized content—help reduce labeling bias and allow us to detect statistically significant improvements”article, stated

Evaluation data inside the primary application

Rejected because: An isolated evaluation service reduces data leakage risk and unauthorized dependencies.

“We established a service dedicated to evaluation data, isolated from Harvey’s primary application to prevent unintentional data leakage or unauthorized dependencies.”article, stated

Patterns

eval-driven-development (evaluation)

Harvey uses expert datasets, A/B comparisons, Likert ratings, model vetting, canaries, and automated graders to decide model and pipeline changes.

“These evaluations give us the signal we need to justify changes to our underlying model stack. In this case, the feedback helped validate our decision to shift more workloads to GPT-4.1, knowing it would lead to a measurable jump in quality for end users.”article, stated
“We run a suite of lightweight canary evaluations nightly to validate the day's code changes before they go to production”article, stated

human-in-the-loop-gate (governance)

Legal experts create datasets, judge answers, and directly shape product decisions.

“Engineers regularly hop on calls with partners at some of the most prestigious law firms in the world.”article, stated
“* A/B preference tests: Experts see two anonymized answers side-by-side, with model order randomized, and choose the better one.”article, stated

llm-as-judge (evaluation)

Automated evaluators grade answers and an LLM performs binary document matching for citations.

“The evaluator uses this information to produce two results: a grade that reflects how well the model’s output meets the expected quality or correctness standards”article, stated
“Finally, an LLM performs a binary document-matching evaluation, confirming whether the retrieved candidate refers to the same document as the original citation.”article, stated

rag-retrieval (retrieval)

Harvey evaluates traditional and agent-based retrieval against expert references and uses retrieved documents to construct model context.

“We then evaluate our retrieval systems—both traditional and agent-based—against these references”article, stated

privacy-preserving-processing (governance)

The evaluation data service isolates sensitive data and enforces row-level role-based access controls.

“Moreover, our fine-grained role-based access control system enforces strict privacy policies at the row level, enabling us to segment data between public, confidential, and restricted tiers.”article, stated
“This means sensitive legal documents and customer-provided examples remain under the tightest restrictions, while higher-level metrics and aggregate statistics can be safely shared more broadly.”article, stated

Tradeoffs

Evaluation mix

Chose expert-led review plus automated evaluation over either expert review or automation alone.

Rationale: The combined design gives legal depth and broader continuous coverage.

“By combining expert-led reviews with automated evaluations, we close critical gaps that either approach might leave on its own”article, stated

Human comparison protocol

Chose randomized, anonymized A/B and Likert evaluations over uncontrolled expert feedback.

Rationale: Structured controls reduce labeling bias and support statistically significant release decisions.

“randomized ordering, standardized prompts, and anonymized content—help reduce labeling bias and allow us to detect statistically significant improvements”article, stated

Evaluation data architecture

Chose isolated dedicated evaluation data service over evaluation data embedded in the primary app.

Rationale: Isolation, access control, and versioning protect confidentiality and reproducibility.

“isolated from Harvey’s primary application to prevent unintentional data leakage or unauthorized dependencies.”article, stated

Gotchas (what bit them)

Retrieval errors compound in legal answer pipelines.

Root cause: Agentic search may repeatedly search, reflect, and refine context before answer generation.

Resolution: Harvey evaluates retrieval separately with expert references, precision, recall, NDCG, retrieval power, and context-window tests.

“In an agentic system, this isn’t a one-time step—it’s an iterative process where the system may search, reflect, and refine its context multiple times before generating a response.”article, stated
“That’s why it’s so important to get this stage right: mistakes here can cascade, compounding in later steps and undermining the final answer.”article, stated

Generative legal outputs are hard to evaluate automatically.

Root cause: Answer quality depends on domain expertise, accuracy, helpfulness, clarity, and legal nuance.

Resolution: Harvey built side-by-side LLM comparisons and Likert scoring for expert reviewers.

“evaluating generative outputs, especially those that require domain expertise to judge, is far harder to automate.”article, stated
“we’ve built an internal tool for side-by-side LLM comparisons, enabling domain experts to assess responses in a structured, unbiased way.”article, stated

Citation verification has messy public-source matching problems.

Root cause: Citation strings may be incomplete, misspelled, partial, or ambiguous across millions of documents.

Resolution: Harvey combines metadata parsing, title-focused embeddings, source context, date filters, and LLM binary matching.

“Metadata Weighting: Properly weighting fields like the document name, date, parties, and publication in situations where the citation is partial or ambiguous.”article, stated
“When metadata is partial—such as when only a title fragment or source identifier is available—an embedding-based retrieval approach with date filters is used”article, stated

Evaluation datasets can drift if mutable during experiments.

Root cause: Changing samples or annotations can make quality improvements look like model improvements.

Resolution: Harvey locks published datasets and versions samples for reproducible comparisons.

“This immutability enhances reproducibility, helping product teams confirm that any observed quality improvements are the result of deliberate changes rather than shifting data sets or annotation drift.”article, stated

Evaluation

The case is itself an evaluation system: expert-created retrieval datasets, precision/recall/NDCG, side-by-side A/B preference tests, 1-7 Likert ratings, statistical significance, nightly canaries, production monitoring, model vetting, rubrics, confidence scores, and citation-matching benchmarks.

Metrics: Precision, recall, and NDCG for retrieval · Retrieval power and model context · A/B preference tests · 1-7 Likert ratings · Mean rating 5.10 to 5.63 · Median rating 5 to 6 · Over 95% citation matching accuracy · Nightly canary evaluations

3 supporting quotes
“using metrics like precision (the proportion of relevant results), recall (how many of the relevant documents were found), and NDCG”article, stated
“The mean rating improved by over 10% (from 5.10 to 5.63), and the median score rose from 5 to 6—a full point higher on a 7-point scale.”article, stated
“This combination of structured parsing, intelligent retrieval, and LLM-based judgment has yielded over 95% accuracy on our internal benchmark dataset validated by attorneys.”article, stated

Outcomes

Harvey reports daily professional usage, expert review across multiple product lines, nightly canary evaluations, a citation corpus of millions of documents, a 5.10 to 5.63 mean rating improvement, a median rating rise from 5 to 6, and over 95% citation-verification accuracy.

3 supporting quotes
“These evaluations give us the signal we need to justify changes to our underlying model stack. In this case, the feedback helped validate our decision to shift more workloads to GPT-4.1, knowing it would lead to a measurable jump in quality for end users.”article, stated
“a seemingly minor change to how we chunk retrieved documents led to a noticeable boost in both perceived helpfulness and factual grounding”article, stated
“This gives us complete control over how data is accessed, updated, and versioned, while still allowing for the flexibility needed to handle anything from small pilot projects to large-scale research efforts.”article, stated