Building and Evaluating Legal AI at Scale with Domain Expert Integration

harvey · Legal · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: legal AI product development and evaluation

Harvey needed to build and evaluate legal AI products where documents are complex, legal quality is nuanced and subjective, mistakes can be career-impacting, and sensitive customer data makes ordinary feedback, bug reports, and benchmark construction difficult.

Constraints:

• lawyers work with hundreds or thousands of pages, scanned notes, handwriting, multi-column layouts, embedded tables, and cross-references
• outputs must include long legal text, complex tables, diagrams, charts, citations, and domain-specific language
• slightly misconstrued legal statements can be more important than obvious hallucinations
• human expert preference remains the highest-quality signal but is slow and expensive
• legal datasets, product feedback, and bug reports are difficult to obtain because customer work is sensitive
• academic legal benchmarks can be too simple and closed-ended for real legal work
• multi-step agentic workflows require component-level evaluation rather than only end-to-end scoring
• domain-specific process knowledge for agentic legal workflows often exists only in conversations or document margins

Architecture

Harvey combines a legal AI product platform with embedded lawyer expertise and a three-layer evaluation stack. The product layer includes a general assistant, document extraction, domain-specific workflows, agentic search, personalization/memory, long-running tasks, firm-specific knowledge, templates, and citations. The development layer embeds lawyers with engineers and product teams through use-case discovery, dataset collection, rubric writing, UI iteration, and end-to-end testing. The evaluation layer combines human preference ratings, domain-expert-authored LLM-as-judge rubrics such as Big Law Bench, component-level evaluation for RAG and workflows, LangSmith routine evals, custom human-rater tooling, product-specific tests, and internal dogfooding before launch.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  legal_professional_user["Legal professional user"] --> harvey_assistant_and_workflows["Harvey assistant and workflows"]
  firm_knowledge_and_templates["Firm knowledge and templates"] --> harvey_assistant_and_workflows
  document_corpus["Document corpus"] --> harvey_assistant_and_workflows
  harvey_assistant_and_workflows --> citation_and_verification_layer["Citation and verification layer"]
  citation_and_verification_layer --> legal_professional_user
  lawyer_in_the_loop_product_team["Lawyer-in-the-loop product team"] --> ai_prototyping_stack["AI prototyping stack"]
  ai_prototyping_stack --> harvey_assistant_and_workflows
  lawyer_in_the_loop_product_team --> human_preference_evaluation_tooling["Human preference evaluation tooling"]
  lawyer_in_the_loop_product_team --> big_law_bench_and_llm_judge["Big Law Bench and LLM judge"]
  harvey_assistant_and_workflows --> component_evaluation_harness["Component evaluation harness"]
  document_corpus --> component_evaluation_harness
  citation_and_verification_layer --> component_evaluation_harness
  component_evaluation_harness --> langsmith_and_custom_eval_tools["LangSmith and custom eval tools"]
  human_preference_evaluation_tooling --> product_specific_tests_and_dogfooding["Product-specific tests and dogfooding"]
  big_law_bench_and_llm_judge --> product_specific_tests_and_dogfooding
  product_specific_tests_and_dogfooding --> lawyer_in_the_loop_product_team

Components

Data flow

Customer legal documents, firm knowledge bases, templates, and workflow context enter Harvey's assistant, document-analysis, and agentic workflow products. Lawyers and product teams build prototypes and evaluation datasets, then collect side-by-side human preference ratings, run Big Law Bench and product-specific automated evaluations, decompose RAG/workflow components such as query rewriting, retrieval, answer generation, and citation creation, and use qualitative dogfooding/customer feedback to decide whether new models, prompts, algorithms, or product changes are ready for customers.

Why this architecture

Harvey's architecture reflects the legal domain's high quality bar: factual correctness alone is insufficient, domain experts must define nuance and taste, and multi-step legal work needs both source-grounded citations and component-level evaluation. The combination of lawyers, prototypes, human preferences, LLM judges, custom benchmarks, component evals, and dogfooding reduces the risk that an apparently strong model fails in a customer-specific legal workflow.

Models used: state-of-the-art LLMs · GPT-4.1 · LLM-as-judge · RAG systems · agentic search · multi-step agents and workflows

Scale

• Harvey has just under 400 customers.
  “We have just under 400 customers on I think all continents maybe except Antarctica at this point.”transcript, stated
• Harvey is used by one-third of the largest 100 US law firms and eight of the largest ten.
  “in the US, onethird of the largest 100 and I think eight out of 10 of the largest 10 law firms use Harvey.”transcript, stated
• Legal document-analysis workflows can involve hundreds or thousands of contracts, documents, or emails.
  “dealing with thousands of contracts or documents, thousands of emails that need to be analyzed”transcript, stated
• Harvey can analyze hundreds or thousands of documents at once and save hours or weeks of work.
  “Harvey can analyze hundreds or thousands of documents at once and output to a table or summarize the results. This literally saves hours, sometimes weeks of work.”transcript, stated
• Human preference ratings use a one-to-seven scale.
  “on a scale of say one to seven, from one being very bad to seven being very good, how would you rate each response?”transcript, stated
• The GPT-4.1 launch example happened in April.
  “in April, OpenAI actually released GPT 4.1.”transcript, stated
• Evaluation tooling paid back about tenfold.
  “In our case, I could say it paid back 10fold.”transcript, stated
• An M&A transaction can take months or years and involve hundreds of subtasks.
  “This is typically many months, sometimes years of work, and it's typically broken down into hundreds of subtasks or projects.”transcript, stated

Alternatives considered (roads not taken)

Rely on model capability without domain expert integration

Rejected because: Legal nuance and user empathy require lawyers to be embedded throughout use-case selection, datasets, rubrics, UI, testing, demos, and feedback translation.

Use PRDs and specs as the main development artifact

Rejected because: Harvey believes legal AI product quality emerges faster through frequent prototypes and iteration with experts.

Depend only on human evaluations

Rejected because: Human evals are the best signal but are time-consuming and expensive, so Harvey supplements them with model-driven and component evals.

Use simple academic legal benchmarks as the main automated eval

Rejected because: LegalBench-style yes/no questions do not reflect real legal work, so Harvey built Big Law Bench for open-ended subjective tasks.

Evaluate complex workflows only end to end

Rejected because: Breaking workflows into components makes evaluation tractable and enables more automated evaluation.

Launch GPT-4.1 after strong benchmark and human-rater results

Rejected because: Additional product-specific tests and dogfooding caught off-brand response regressions before customer rollout.

Patterns

human-in-the-loop-gate (domain expert integration)

Harvey embeds lawyers throughout product development and uses trained legal experts as raters, rubric authors, dogfooders, demo participants, and feedback translators.

eval-driven-development (legal quality evaluation)

Harvey ties product development to human preference evals, Big Law Bench, LLM-as-judge rubrics, product-specific datasets, component evals, dogfooding, and launch decisions.

rag-retrieval (legal corpus QA)

The canonical component-eval example decomposes RAG over legal corpora into query rewriting, chunk/document retrieval, answer generation, and citation creation.

structured-output-contract (document analysis outputs)

Harvey's workflows generate tables, structured analysis, citations, and rubric-scored outputs, and document extraction can output table results.

observability-tracing (evaluation tooling)

Harvey uses LangSmith extensively for routine evaluation of broken-down tasks while combining it with custom human-rater tools.

agent-memory (personalization)

Harvey describes agentic capabilities that include personalization and memory alongside multi-step search and long-running tasks.

Tradeoffs

Embed lawyers throughout development.

Chose Lawyer-in-the-loop product development over Engineering-led AI development with domain review only at the end.

Rationale: Legal products require domain expertise and user empathy in use cases, datasets, rubrics, UI, testing, demos, and feedback.

Iterate through prototypes.

Chose Prototype over PRD over Spec-first product requirements documents.

Rationale: Prototypes make legal AI workflows tangible and help lawyers, engineers, and product teams iterate on real outputs.

Use human ratings for launch decisions.

Chose Side-by-side preference, 1-7 ratings, and qualitative expert feedback over Automated model metrics alone.

Rationale: Legal quality is nuanced and subjective enough that expert preference remains the highest-quality signal.

Build a legal-work benchmark.

Chose Big Law Bench with open-ended subjective tasks over LegalBench-style yes/no questions.

Rationale: Real legal work requires analysis over external materials and paragraphs of subjective text, not simple classification.

Decompose workflow evaluation.

Chose Component-level evaluation for RAG, workflows, and citations over Only evaluating final workflow output.

Rationale: Breaking each step into an eval unit makes multi-step legal agents more tractable to automate and debug.

Gotchas (what bit them)

Legal correctness is not enough to capture quality.

Root cause: Two answers can both be factually correct and non-hallucinatory, while lawyers prefer one because of nuance and definition detail.

Resolution: Use domain-expert human preference judgments and qualitative feedback in addition to automated metrics.

Sensitive legal work makes datasets and feedback difficult to obtain.

Root cause: Customer legal work is confidential, limiting reliable datasets, bug reports, and product feedback.

Resolution: Invest in in-house domain experts, internal dogfooding, custom tooling, and careful evaluation workflows.

Academic legal benchmarks can be too simple.

Root cause: Real legal work is open-ended, context-rich, and subjective, unlike yes/no questions with no external material.

Resolution: Build Big Law Bench with complex open-ended legal tasks and domain-expert-crafted rubrics.

Human evals do not scale cheaply.

Root cause: The highest-quality legal evaluations require trained attorneys and nuanced review.

Resolution: Improve rater-tool throughput and supplement human evals with model-driven and component-level evaluations where possible.

Model launch gates can miss product-style regressions.

Root cause: A model can score well on benchmark and rater evaluations while adopting off-brand wording or other qualitative regressions.

Resolution: Run product-specific datasets and internal dogfooding before customer rollout.

The process data needed for legal agents often has not been captured.

Root cause: Complex legal work is learned through projects, conversations, and marginalia rather than formal playbooks.

Resolution: Treat process-data extraction as a future foundation for better domain-specific agentic workflows.

Evaluation

Harvey evaluates through side-by-side expert ratings, 1-7 absolute scores, qualitative comments, Big Law Bench, LLM-as-judge rubrics crafted by in-house lawyers, component-level RAG/workflow/citation evals, LangSmith routine evals, product-specific datasets, and internal dogfooding before customer rollout.

Metrics: just under 400 customers · one-third of the largest 100 US law firms · eight out of the largest ten US law firms · hundreds or thousands of documents analyzed at once · 1-7 human rating scale · GPT-4.1 evaluated before customer rollout · evaluation tooling paid back about tenfold · M&A process data can span months or years and hundreds of subtasks

Outcomes

• Harvey serves nearly 400 customers, including major US law firms, with legal AI assistants, document analysis, and domain-specific workflows.
• Their lawyer-in-the-loop development process has produced dozens of workflows through rapid prototyping and expert iteration.
• Big Law Bench gives Harvey a more realistic automated benchmark for open-ended legal tasks than simple academic legal benchmarks.
• The GPT-4.1 launch process combined benchmark, human-rater, product-specific, and dogfooding evaluations and caught off-brand response regressions before customer rollout.
• Evaluation tooling improved iteration speed, product quality, confidence, and shipping speed.
• Harvey identifies process-data capture as a likely next frontier for domain-specific legal agents.

The source cites just under 400 customers, one-third of the top 100 US law firms, eight of the top ten law firms, hundreds or thousands of documents analyzed at once, a 1-7 rating scale, GPT-4.1 launch evaluation, tenfold evaluation-tooling payback, and M&A workflows lasting months or years across hundreds of subtasks.