Large-Scale Legal RAG Implementation with Multimodal Data Infrastructure

harvey_/_lance · Legal · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: High-stakes legal RAG over multimodal corpora

Harvey needed retrieval infrastructure for legal AI workflows where queries are expert, multi-part, jurisdiction-specific, and high stakes. The system had to span on-demand document uploads, matter-specific vaults, and country-scale legal corpora; combine semantic, keyword, filtering, and re-ranking behavior; support sensitive legal data with privacy and retention controls; and make retrieval quality measurable enough for rapid iteration.

Constraints:

• legal tasks include drafting, document analysis, and legal workflows
• data ranges from 1-50 uploaded documents to vaults and country-scale corpora
• large corpora can contain tens of millions of long, dense legal documents
• queries mix semantic intent, dates, directives, article references, keywords, abbreviations, and implicit filters
• legal data can include confidential deals, IPOs, financial filings, and customer-specific retention constraints
• online query latency and offline ingestion, reingestion, and ML experimentation both matter
• generic retrieval approaches are insufficient without domain experts, evaluation, and flexible multimodal data infrastructure

Architecture

Harvey combines a legal AI assistant workflow with LanceDB's AI-native multimodal lakehouse. Harvey's side handles legal RAG across three data scales: small uploads, project vaults, and large knowledge corpuses. Retrieval development uses domain experts, legal query analysis, sparse/dense retrieval, filtering, heuristics, LLM processing, and a multi-tier evaluation strategy. LanceDB provides a unified storage and retrieval layer on object store for images, videos, audio, embeddings, text, tabular data, and time series data in one table. It supports offline preprocessing/training and online serving, compute-storage separation, multiple vector columns, vector plus full-text search, re-ranking, GPU indexing at billion-vector scale, and Lance format for random access, scans, blob data, schema evolution, and compatibility with Arrow, Spark, Ray, PyTorch, pandas, and Polars.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  assistant["Harvey legal AI assistant"] --> uploads["Small upload workspace"]
  assistant --> vaults["Project vaults"]
  assistant --> retrieval["Retrieval orchestration layer"]
  corpora["Legal corpora"] --> retrieval
  experts["Domain expert workflow"] --> corpora
  experts --> evals["Evaluation framework"]
  retrieval --> lance["LanceDB lakehouse"]
  lance --> format["Lance format"]
  lance --> gpu["GPU indexing"]
  lance --> privacy["Privacy and retention controls"]
  vaults --> privacy
  retrieval --> evals
  evals --> assistant

Components

Data flow

Legal data enters as uploaded documents, matter vaults, or country-scale corpora. Domain experts help interpret structure, jurisdictions, explicit query fields, and implicit query patterns. Harvey uses heuristics and LLM processing to categorize and preprocess the data, then retrieval combines semantic search, exact keyword matching, filtering, vector/full-text search, and re-ranking. LanceDB stores multimodal source data, embeddings, text, scalar fields, and blobs together on object storage, enabling search, analytics, training, preprocessing, and feature engineering from a single source of truth. Evaluation runs at multiple fidelity levels: expert reviews, expert-labeled criteria, synthetic or automated checks, and quantitative precision/recall and deterministic retrieval criteria.

Why this architecture

Harvey needs hybrid retrieval and domain-specific processing because legal queries combine semantic meaning, exact citations, temporal filters, multiple regulations, and specialized jargon. LanceDB's lakehouse architecture fits because legal RAG is not only online vector search; teams also need preprocessing, analytics, training, feature extraction, schema evolution, random access, and multimodal data in one source of truth. The evaluation-heavy operating model is necessary because engineers cannot reliably judge legal correctness without expert feedback and retrieval-specific metrics.

Models used: large language models · retrieval-augmented generation · dense retrieval · sparse retrieval · vector search · full-text search · re-ranking · LLM processing techniques · LanceDB · Lance format

Scale

• The smallest Harvey context handles roughly 1-50 uploaded documents.
  “that's like a smaller 1 to 50 range.”transcript, stated
• Large legal corpora can contain tens of millions of documents.
  “one of these corpuses can be like yeah tens of millions of docs.”transcript, stated
• LanceDB reports indexing roughly three to four billion vectors in under two to three hours.
  “three or four billion vectors in a single table um that can in index in under uh two or three hours.”transcript, stated
• The system must support both online low-latency querying and offline ingestion, reingestion, and ML experimentation.
  “we need very good performance both online and offline. online being like querying over this. You want good latency”transcript, stated

  “offline being like ingestion, reingestion, running ML experiments for different variations and such.”transcript, stated
• The legal data includes confidential deals, IPOs, and financial filings.
  “A lot of this data is sensitive for like confidential deals or confidential I don't know IPOs, financial filings”transcript, stated

Alternatives considered (roads not taken)

Use only generic semantic vector search

Rejected because: Legal queries also require implicit filters, exact regulation IDs, multi-part legal references, and domain jargon handling.

Treat retrieval quality as primarily an algorithm problem

Rejected because: Harvey says the team spends more effort validating systems than inventing algorithms or agentic techniques.

Store AI data across multiple specialized systems

Rejected because: LanceDB argues split copies force AI teams to spend effort keeping data glued together and synchronized.

Use traditional Parquet/Iceberg-style storage alone for multimodal AI data

Rejected because: The speaker says those formats miss features needed for random access, blob data, and schema evolution in multimodal AI workloads.

Patterns

rag-retrieval (retrieval)

Harvey uses retrieval-augmented legal AI over uploaded documents, vaults, and country-scale legal corpora.

hybrid-search (retrieval)

The retrieval layer combines semantic matching, exact keyword matches, filters, multiple vectors, full-text search, and re-ranking.

eval-driven-development (evaluation)

Harvey uses expert reviews, expert-labeled criteria, automated evaluation, precision/recall, and deterministic criteria to guide legal RAG iteration.

privacy-preserving-processing (security)

The system accounts for sensitive legal data, customer segregation, and document retention limits.

retrieval-ranking (retrieval)

LanceDB exposes re-ranking on top of vector and full-text retrieval for complex legal query patterns.

data-quality-loop (data)

Domain experts and automation are used together to categorize, organize, preprocess, and evaluate legal data so retrieval can improve.

Tradeoffs

Evaluation strategy

Chose Tiered expert and automated evaluations over Only expert review or only automated metrics.

Rationale: Expert reviews are highest fidelity but expensive, while automated metrics support faster iteration.

Data architecture

Chose Multimodal lakehouse on object store over A vector database alone.

Rationale: The application needs search plus analytics, training, preprocessing, feature extraction, summaries, text descriptions, and multimodal storage.

Retrieval API surface

Chose Combined vector columns, full-text search, and re-ranking over Single-vector similarity search only.

Rationale: Legal query patterns need exact matches, semantic matches, filters, and dynamic navigation over complex data.

Storage model for AI data

Chose One table with mixed blobs, embeddings, text, and scalar data over Separate synchronized copies across multiple systems.

Rationale: Keeping data in one place avoids glue code and synchronization work while supporting search, analytics, training, and preprocessing.

Gotchas (what bit them)

Legal questions rarely map to one retrieval mode.

Root cause: A single query can combine semantic intent, dates, specialized datasets, exact legal IDs, multiple regulations, and domain abbreviations.

Resolution: Break down the query and use the appropriate retrieval and filtering technologies for each part.

Engineers cannot rely on intuition alone in specialized legal domains.

Root cause: Legal documents and queries contain domain-specific details that engineers or researchers may not inherently understand.

Resolution: Work with domain experts and invest in expert-driven evaluation procedures.

High-fidelity legal evaluation is expensive.

Root cause: The highest quality feedback requires experts to review outputs, analyze them, and write reports.

Resolution: Use a tiered evaluation strategy that balances expert review with curated criteria and automated quantitative metrics.

Multimodal AI data fragments across systems if storage is not designed for it.

Root cause: Traditional formats can lack random access, blob support, and efficient schema evolution for mixed embeddings, text, images, video, and scalar data.

Resolution: Use Lance format and a lakehouse architecture that stores multimodal data together as a single source of truth.

Privacy requirements affect the retrieval infrastructure itself.

Root cause: Legal customers may require segregated storage and legally constrained retention periods.

Resolution: Build flexible data privacy and retention capabilities into the database/infrastructure layer.

Evaluation

Harvey evaluates legal RAG with a layered approach: expert legal reviews and reports for highest fidelity, expert-labeled criteria that can be evaluated synthetically or automatically, and faster quantitative checks such as retrieval precision and recall, folder correctness, section correctness, and keyword matching.

Metrics: expert review quality · expert-labeled evaluation criteria · synthetic or automated evaluation over criteria · retrieval precision · retrieval recall · document folder accuracy · section correctness · keyword matching validation · online latency · offline ingestion and reingestion performance

Outcomes

• Harvey and LanceDB describe a legal RAG stack spanning small uploads, matter vaults, and country-scale corpora.
• The architecture supports multimodal AI data in one lakehouse table, with vector/full-text search, re-ranking, GPU indexing, and compute-storage separation.
• The system design emphasizes evaluation-driven iteration over purely algorithmic development.
• LanceDB reports billion-scale GPU indexing, with a stated record around three to four billion vectors indexed in under two to three hours.
• The source is a technical talk and does not provide production accuracy, latency, or customer-impact numbers.

The talk describes 1-50 document upload contexts, large matter vaults, country-scale corpora, tens of millions of documents per corpus, sensitive legal datasets, online and offline workloads, and LanceDB GPU indexing around three to four billion vectors in under two to three hours.