Agentic Search for Multi-Source Legal Research Intelligence

harvey · Legal · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal research

Harvey needed retrieval that could follow the iterative reasoning loops lawyers use across statutes, case law, internal documents, and proprietary legal sources.

Constraints:

• legal accuracy
• privacy guarantees
• multi-source citation-backed answers

Architecture

Harvey built Agentic Search as a ReAct-style retrieval loop for legal research: the agent plans the query, selects among legal knowledge sources, retrieves and reasons over results, checks whether the context is sufficient, and only then synthesizes a citation-backed response. The loop sits beside a privacy-preserving evaluation flywheel where legal experts create synthetic queries, OpenTelemetry traces feed LangSmith offline evals, and improvements tune tool descriptions, prompts, and source-selection signals.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  query-planner[Query planner] --> tool-selector-and-retriever[Tool selector and retriever]
  tool-selector-and-retriever --> knowledge-sources[Knowledge sources]
  tool-selector-and-retriever --> reasoning-synthesizer[Reasoning synthesizer]
  reasoning-synthesizer --> completeness-checker[Completeness checker]
  completeness-checker --> tool-selector-and-retriever
  completeness-checker --> citation-synthesizer[Citation synthesizer]
  reasoning-synthesizer --> citation-synthesizer
  citation-synthesizer --> knowledge-sources
  evaluation-flywheel[Evaluation flywheel] --> tool-selector-and-retriever

Components

Data flow

A legal user submits a complex research prompt. The query planner identifies what information is needed and which legal sources are relevant. The tool-selection layer formulates source-specific searches across Harvey knowledge sources, Vault/iManage documents, and LexisNexis case law. The reasoning step connects retrieved facts to legal standards, the completeness checker loops back for more retrieval when gaps remain, and the final synthesizer writes a source-cited answer.

Why this architecture

The architecture is a loop because legal research is iterative and source-dependent: one-shot RAG cannot adjust strategy, pick sources, or know when context is sufficient. The evaluation layer exists because Harvey cannot inspect customer queries, so expert-generated queries and telemetry-backed offline evals substitute for direct production review.

Models used: LLMs · OpenAI Agent SDK instrumentation

Scale

• The retrieval layer spans Harvey knowledge, customer document stores, and legal databases rather than a single corpus.
  “Scalable knowledge: Agents tap into Harvey's 150+ legal knowledge sources, internal Vault documents, iManage integration, and proprietary databases like case law knowledge through LexisNexis.”article, stated
• Source-selection precision improved from near zero to 0.8-0.9 after the evaluation loop gave the agent clearer source-use signals.
  “This approach solved our source selection challenge by giving the agent clearer signals about when to use each knowledge source — improving tool selection precision from near zero to 0.8-0.9.”article, stated
• Complex legal questions now scale retrieval effort to 3-10 operations instead of stopping after one call.
  “Complex queries that initially resolved in a single tool call now appropriately scale to 3-10 retrieval operations based on query demands.”article, stated

Alternatives considered (roads not taken)

Static one-shot retrieval

Rejected because: It cannot adapt as discoveries change the search strategy, decide relevant sources, or determine context sufficiency.

Direct human review of production customer queries

Rejected because: Harvey privacy guarantees prevent humans from accessing real customer queries, so improvement must use synthetic expert queries and aggregate feedback.

Fixed one-or-two-tool-call agent behavior

Rejected because: The team found simple and complex queries need different effort levels; evaluation calibrated retrieval operations to query demands.

Patterns

rag-retrieval (retrieval)

The system expands naive RAG into source-aware legal research over Harvey knowledge, internal files, and legal databases.

eval-driven-development (quality)

Legal experts created synthetic eval queries and offline traces to tune source selection and retrieval depth without exposing customer queries.

Tradeoffs

How to improve production search under strict privacy limits.

Chose synthetic expert-generated evaluation over human review of real customer queries.

Rationale: The privacy model prevented direct inspection of real queries, so improvement had to come from expert-created proxies and telemetry.

Retrieval effort calibration.

Chose adaptive retrieval that scales from minimal lookup to multiple rounds over a fixed one-or-two-tool-call budget for nearly every query.

Rationale: Simple lookups should stay cheap, but complex legal research needs enough retrieval rounds to gather and reconcile complete context.

Search architecture for professional legal research.

Chose an iterative agentic search loop over static one-shot retrieval.

Rationale: The loop mirrors how lawyers refine research, cross-check sources, and dig deeper before producing a cited answer.

Gotchas (what bit them)

The first agent treated simple and complex questions too similarly.

Root cause: It lacked calibrated effort for when a query demanded exhaustive multi-source coverage.

Resolution: Use eval data to calibrate effort so complex legal questions trigger more retrieval operations.

Evaluation

Harvey tracked hallucination, tool recall, retrieval recall, formatting, answer quality, and source-selection precision in offline evals.

Metrics: tool selection precision 0.8-0.9 · 3-10 retrieval operations for complex queries · hallucination · answer quality

Outcomes

• Agentic Search raised baseline Assistant query quality and became the technical foundation for Deep Research.

The source describes broad product use across legal knowledge sources rather than only one workflow.