Enterprise-Grade RAG Systems for Legal AI Platform

harvey · Legal · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI retrieval

Harvey needed enterprise-grade RAG for privacy-conscious legal and professional-services use cases across uploaded files, long-term vault projects, and third-party legal databases while preserving latency, accuracy, ingestion speed, security, and evaluation quality.

Constraints:

Legal data is sensitive and must stay isolated.
Use cases span thread-level files, long-term vault projects, and third-party legal databases.
Users query documents immediately, so ingestion speed matters.
Large corpora need high indexing throughput, high accuracy, and low query latency.
Public ground-truth evaluation sets do not exist for specialized legal tasks.

Architecture

Harvey's RAG architecture uses vector databases to search user-uploaded files, Vault projects, and third-party legal sources. The platform evaluates vector databases for scalability, metadata filtering, privacy, hosting, ingestion throughput, and latency, primarily using LanceDB Enterprise in production while using Postgres/PGVector for smaller public-data projects.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-uploaded-files["User-uploaded files"] --> vector-database["Vector database"]
  vault-projects["Vault projects"] --> vector-database
  third-party-legal-databases["Third-party legal databases"] --> vector-database
  vector-database --> semantic-retrieval["Semantic retrieval"]
  semantic-retrieval --> llm-answer-workflow["LLM answer workflow"]
  llm-answer-workflow --> domain-expert-evaluation["Domain expert evaluation"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User-uploaded files`	Thread-scoped legal documents that persist for the duration of a conversation.	`Vector database`
`Vault projects`	Long-term project document collections.	`Vector database`
`Third-party legal databases`	Regulations, laws, statutes, and public/private legal data for research and memos.	`Vector database`
`Vector database`	Indexes embeddings and source data with privacy, latency, ingestion, and metadata requirements.	`Semantic retrieval`
`Semantic retrieval`	Finds relevant legal context by vector similarity.	`LLM answer workflow`
`LLM answer workflow`	Generates grounded legal-assistance responses from retrieved context.	`Domain expert evaluation`
`Domain expert evaluation`	Guides, evaluates, and refines models for open-ended legal tasks.	—

Data flow

User or project documents are isolated, embedded, indexed in an enterprise vector database, searched semantically for relevant legal context, and fed to LLM workflows whose outputs are evaluated with domain-expert feedback.

“At Harvey we build retrieval systems for three key types of data sources that reside in our platform:”article, stated

“RAG is powered by semantic search, which uses embedding vectors to capture semantic meaning of text or images and find relevant items for the user.”article, stated

Why this architecture

Harvey centers the design on vector database selection because legal RAG must combine high-scale semantic search with privacy, customer-controlled storage, ingestion speed, metadata filtering, and domain-expert evaluation.

“At the heart of RAG systems are vector databases (VectorDBs), which enable efficient searching across massive datasets.”article, stated

“Choosing the right database requires careful consideration of performance and compliance.”article, stated

Models used: LLMs · RAG · LanceDB Enterprise · Postgres with PGVector · Vector databases · Embedding vectors

Scale

User-uploaded file sets range from 1 to 50 documents.
“User-uploaded files in Assistant that persist for the duration of a thread (1-50 documents)”article, stated
Vault projects can contain 1,000 to 10,000 documents.
“User-stored documents in Vault projects persisted for the duration of a long-term project (1,000-10,000 documents)”article, stated
Harvey serves users in 45 countries.
“Since we now serve users in 45 countries, these documents can be in all different languages and regional formats.”article, stated
The article says enterprise RAG must search over millions of documents quickly.
“Large corpora require optimized indexes and infrastructure that can search over millions of documents quickly”article, stated

Alternatives considered (roads not taken)

Use Postgres/PGVector for all workloads.

Rejected because: Harvey primarily uses LanceDB Enterprise in production for latency, accuracy, throughput, scale, privacy, and hosting.

“We primarily use LanceDB Enterprise in production because of its strength across latency, accuracy, ingestion throughput, scalability, data privacy, and hosting.”article, stated

Store customer embeddings and source data outside customer-controlled environments.

Rejected because: Harvey requires private cloud hosting and customer-controlled storage for enterprise deployments.

“store both embeddings and source data in our customers’ own secure storage (e.g., their private cloud buckets).”article, stated

Evaluate legal RAG against public ground-truth datasets.

Rejected because: The specialized tasks lack straightforward public ground-truth sets.

“Straightforward ground-truth evaluation sets do not exist publicly for these specialized tasks.”article, stated

Patterns

`rag-retrieval` (retrieval)

Harvey uses vector-search-backed RAG to ground legal AI answers in uploaded, vault, and third-party data.

“Retrieval Augmented Generation (RAG), which enriches LLMs with relevant, up-to-date or user-inputed knowledge”article, stated

`privacy-preserving-processing` (governance)

Sensitive user documents are isolated and stored in customer-controlled secure storage.

“sensitive user documents need to be strictly isolated.”article, stated

`eval-driven-development` (evaluation)

Harvey works with domain experts to define evaluation objectives and refine retrieval models.

“working hand-in-hand with domain experts to develop, evaluate, and refine our models.”article, stated

Tradeoffs

Use LanceDB Enterprise in production.

Chose LanceDB Enterprise for latency, accuracy, throughput, scale, privacy, and hosting. over Postgres/PGVector for every workload..

Rationale: Harvey keeps Postgres for smaller public-data projects but chooses LanceDB for production enterprise requirements.

“we still appreciate the simplicity and feature-completeness of Postgres for rapid development, and use it for smaller-scale projects”article, stated

Require private cloud/customer storage.

Chose Customer-controlled embeddings and source data. over Centralized vendor-controlled storage..

Rationale: Legal customers require stringent security and privacy.

“This setup is crucial for meeting stringent security and privacy requirements”article, stated

Gotchas (what bit them)

Legal RAG has no easy public ground truth.

Root cause: Legal tasks are specialized, open-ended, and domain-specific.

Resolution: Use domain experts to guide data sources, usage feedback, and evaluation objectives.

“We need to work with domain experts to evaluate these models and develop evaluation objectives”article, stated

Data privacy extends to embeddings.

Root cause: Vectors can encode sensitive customer document information.

Resolution: Host vector databases privately and store embeddings/source data in customer-controlled storage.

“This same standard extends to the vectors that we embed on our customers' documents.”article, stated

Immediate querying conflicts with ingestion complexity.

Root cause: Users start querying documents immediately after upload, but large corpora need indexing and isolation.

Resolution: Optimize for ingestion speed and high indexing throughput.

“Users will also start querying documents immediately, so ingestion speed is critical.”article, stated

Evaluation

Harvey evaluates vector databases and retrieval systems by scalability, ingestion throughput, query latency, metadata filtering, privacy, accuracy, and domain-expert feedback.

Metrics: Ingestion throughput · Query latency · Accuracy · Scalability · Metadata filtering · Privacy/enterprise readiness

“When evaluating vector databases for RAG, we typically focus on:”article, stated

Outcomes

Harvey primarily uses LanceDB Enterprise in production.
Harvey still uses Postgres/PGVector for smaller-scale public-data projects.
The architecture supports legal retrieval across user files, vault projects, and third-party legal data.

The article cites 45 countries, 1-50 thread documents, 1,000-10,000 vault documents, and millions-document search requirements.

“enabling us to build a product that brings highly accurate answers to hundreds of professional service firms worldwide.”article, stated