Building Custom Cloud Agent Infrastructure for Legal AI at Scale

harvey · Legal · 2026 · source Agentic: yes · promotional bias: high · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI cloud-agent infrastructure

Harvey needed to run large-scale legal agents over privileged client data while satisfying law-firm requirements that managed frontier-lab or cloud-provider runtimes could not meet: multi-model flexibility, true zero data retention, conflict-aware governance, and cost optimization for workflows spanning huge document sets.

Constraints:

Architecture

Harvey built its own cloud-agent runtime instead of relying solely on managed agent platforms. The application layer sends legal work to a provider-normalizing abstraction layer that hides differences in tool-call formats, stop conditions, streaming behavior, failure modes, and sandboxes. A model router uses conflict policy, LAB benchmark results, cost, quality thresholds, and workload characteristics to pick a model or runtime. Agent execution runs inside Harvey's security boundary with session-scoped working memory, intermediate files, tool results, checkpoints, and sandbox compute that are purged after the workflow. The runtime also supports legal-specific governance such as data residency, sovereign deployments, inspectable action records, and conflict-aware model allowlists.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  legal-workflow-request[Legal workflow request] --> conflict-aware-governance[Conflict-aware governance]
  conflict-aware-governance --> model-router[Model router]
  conflict-aware-governance --> data-residency-controls[Data residency controls]
  model-router --> lab-benchmark[LAB benchmark]
  model-router --> provider-abstraction-layer[Provider abstraction layer]
  model-router --> cost-optimizer[Cost optimizer]
  provider-abstraction-layer --> frontier-lab-runtimes[Frontier lab runtimes]
  provider-abstraction-layer --> cloud-provider-runtimes[Cloud provider runtimes]
  provider-abstraction-layer --> self-hosted-open-models[Self-hosted open models]
  provider-abstraction-layer --> legal-agent-orchestrator[Legal agent orchestrator]
  legal-agent-orchestrator --> session-scoped-state[Session-scoped state]
  legal-agent-orchestrator --> ephemeral-sandbox[Ephemeral sandbox]
  legal-agent-orchestrator --> inspectable-action-record[Inspectable action record]
  session-scoped-state --> zero-retention-purge[Zero-retention purge]
  ephemeral-sandbox --> zero-retention-purge
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Legal workflow request Matter-level request such as data-room review or issue-list generation. Conflict-aware governance
Conflict-aware governance Policy layer encoding which models a client or matter is allowed to use. Model router, Data residency controls
Model router Routing layer that chooses the most efficient allowed model meeting the quality threshold. LAB benchmark, Provider abstraction layer, Cost optimizer
LAB benchmark Legal AI benchmark used to evaluate model performance by practice area and task type. —
Cost optimizer Optimization logic for routing away from unnecessary frontier-model usage and tuning sandbox execution. —
Provider abstraction layer Interface that normalizes provider harnesses, tool formats, stop conditions, streaming, failures, and sandboxes. Frontier lab runtimes, Cloud provider runtimes, Self-hosted open models, Legal agent orchestrator
Frontier lab runtimes Managed runtimes from model labs that can offer newest models but may create lock-in or lack ZDR. —
Cloud provider runtimes Model-flexible managed runtimes that may lag newest models or have reliability limits. —
Self-hosted open models Open-source or hosted-by-Harvey models used when they meet quality thresholds at lower cost. —
Legal agent orchestrator Runtime that manages model/tool calls, file loading, parallelization, compute sizing, and state for legal workflows. Session-scoped state, Ephemeral sandbox, Inspectable action record
Session-scoped state Working memory, intermediate files, tool results, and checkpoints that are kept only for the session. Zero-retention purge
Ephemeral sandbox Execution environment optimized for loading files, parallelizing work, and sizing compute for legal workloads. Zero-retention purge
Zero-retention purge Lifecycle control that removes session state after execution to preserve ZDR guarantees. —
Data residency controls Jurisdictional and sovereign-deployment controls for where legal matters may run. —
Inspectable action record Complete record of agent actions for work-product and privilege review. —

Data flow

A legal workflow such as data-room review enters Harvey's cloud-agent platform with matter-specific constraints. Governance policy determines which model providers are allowed for that client or matter, then the routing layer compares model quality and cost using LAB benchmark data. The selected provider or self-hosted model is invoked through the abstraction layer, while the legal agent orchestrator manages hundreds of model and tool calls, loads files, parallelizes work, sizes compute, and maintains temporary session state. During execution, working memory, intermediate files, tool results, and checkpoints remain inside Harvey's boundary and are purged when the session ends. Audit and inspectability requirements preserve an action record without relying on third-party retained state.

4 supporting quotes
“newest models and face redundancy and uptime limits. Neither is sufficient alone, which is why Harvey works with all of them and routes across them. To enable this routing, Harvey built an abstraction layer that normalizes the different agent harnesses, sandboxes, and”summary, stated
“normalizes the different agent harnesses, sandboxes, and behavioral differences across providers. Each provider exposes different tool-call formats, stop conditions, streaming behavior, failure modes, and execution sandboxes, and the same task tuned for one”summary, stated
“runtime, Harvey ensures the agent's entire lifecycle runs inside their security boundary. State is scoped to the session and purged afterward, so the zero-retention guarantee covers the whole workflow rather than just the final model call. This architectural decision”summary, stated
“managed platforms don't currently expose sufficiently. Owning the runtime allows Harvey to route each task to the most efficient model that meets the quality threshold, including open-source models they host themselves. It also enables optimization of the sandbox—how files are loaded, how”summary, stated

Why this architecture

Harvey chose runtime ownership because the legal market turns ordinary agent-platform tradeoffs into hard requirements. Managed lab runtimes create model and orchestration lock-in, and managed state persistence conflicts with zero data retention. Cloud runtimes improve model flexibility but can lag newest models or expose uptime/redundancy limits. Owning the runtime lets Harvey keep state inside its boundary, purge it after sessions, normalize providers, route by matter-specific governance, use smaller or open-source models when tasks are intelligence-saturated, and optimize sandboxes around legal workloads.

4 supporting quotes
“these requirements remain hard blockers that necessitate owning the runtime infrastructure. ## Multi-Model Requirements and Flexibility Harvey's first major requirement is multi-model support, which the company argues is becoming table stakes rather than an”summary, stated
“Automatic state persistence and zero retention are mutually exclusive—you cannot have both. By owning the runtime, Harvey ensures the agent's entire lifecycle runs inside their security boundary. State is scoped to the session and purged afterward, so the”summary, stated
“will underperform on another. The abstraction layer presents a single interface above which the choice of model becomes just a routing decision. ## Zero Data Retention Architecture Every law firm and enterprise contract Harvey signs requires zero data retention (ZDR),”summary, stated
“cheapest, and fastest for the specific task. Achieving this optimization at scale requires fine-grained control over both model routing and the execution sandbox, which managed platforms don't currently expose sufficiently. Owning the runtime allows Harvey to route each task to the”summary, stated

Models used: Anthropic Claude Managed Agents · OpenAI offerings · AWS runtimes · Microsoft Foundry · Google runtimes · Open-source models · GPT-5.4 grader

Scale

Alternatives considered (roads not taken)

Use frontier-lab managed agent runtimes.

Rejected because: They tie customers to one lab's models, create maximum lock-in, and do not offer the zero-retention architecture Harvey needs.

“distinguishes between two types of managed runtimes: frontier labs' runtimes tie customers to that lab's models (maximum lock-in), while cloud providers' runtimes are model-flexible but tend to lag on newest models and face redundancy and uptime limits. Neither is sufficient alone, which is”summary, stated
“the data in question is privileged and confidential and cannot sit on third-party servers. Frontier labs' managed runtimes don't offer ZDR, which means running client matters through them would leave client data retained on the lab's infrastructure—a complete”summary, stated

Use only cloud-provider managed runtimes.

Rejected because: Cloud runtimes are more model-flexible but may lag newest models and have redundancy or uptime limits.

“cloud providers' runtimes are model-flexible but tend to lag on newest models and face redundancy and uptime limits.”summary, stated

Store state during execution and delete it afterward.

Rejected because: Retention followed by deletion is not zero data retention; true ZDR requires never writing customer data to persistent storage.

“There is a tempting shortcut of storing data during a run and calling a deletion endpoint afterward, but this constitutes retention followed by deletion, not zero retention. True ZDR means data is never written to persistent storage in the first place, which”summary, stated
“tempting shortcut of storing data during a run and calling a deletion endpoint afterward, but this constitutes retention followed by deletion, not zero retention. True ZDR means data is never written to persistent storage in the first place, which is an architectural property of the”summary, stated

Route all legal work to the strongest frontier model.

Rejected because: Many tasks are intelligence-saturated, so using the largest model overspends without improving quality enough to justify cost.

“agents that work, but for agents that are economical. The key insight Harvey leverages is that for most tasks, the largest model is no longer necessary. As models have improved, a growing share of legal work has become "intelligence-saturated"—the task sits well within reach”summary, stated
“within reach of a small or open-source model, and using a top frontier model simply overspends capability the task doesn't require. Harvey's LAB benchmark demonstrates that across many task types, open-source models match frontier quality at a fraction of the cost. The goal has shifted from”summary, stated

Patterns

model-routing (operations)

Harvey routes each legal task to an allowed model that meets a quality threshold, using multi-model support to avoid conflicts, lock-in, and overuse of frontier models.

“route each task to the most efficient model that meets the quality threshold”summary, stated
“tuned for one model will underperform on another. The abstraction layer presents a single interface above which the choice of model becomes just a routing decision. ## Zero Data Retention Architecture Every law firm and enterprise contract Harvey signs requires zero data retention (ZDR), which”summary, stated

cost-aware-routing (operations)

The runtime routes intelligence-saturated legal tasks to smaller, open-source, or self-hosted models where they meet quality thresholds, reducing cost versus frontier-only execution.

“open-source models match frontier quality at a fraction of the cost.”summary, stated
“empirically seeing 3-5x cost reductions versus a frontier-only approach”summary, stated

safety-policy-enforcement (safety)

The runtime enforces legal security constraints by keeping data inside Harvey's boundary, scoping state to the session, purging it afterward, and encoding model-provider conflicts.

“the agent's entire lifecycle runs inside their security boundary.”summary, stated
“conflict-aware governance that encodes which models a matter is allowed to use”summary, stated

tool-use-orchestration (architecture)

Harvey's legal agents orchestrate hundreds of model and tool calls across documents, files, sandboxes, and model providers.

“A single agent run can involve hundreds of model and tool calls over a large corpus”summary, stated
“how files are loaded, how work is parallelized, how compute is sized”summary, stated

eval-driven-development (evaluation)

Harvey uses LAB benchmark results by practice area and task type to drive model selection and routing decisions.

“Harvey's LAB benchmark demonstrates that across many task types, open-source models match frontier quality at a fraction of the cost.”summary, stated
“models across practice areas and task types with rubrics graded by GPT-5.4 (mentioned in the figure caption), enabling data-driven routing decisions. This suggests a continuous evaluation pipeline that feeds into their routing logic, allowing them to update routing strategies as model capabilities”summary, stated

observability-tracing (operations)

The legal-specific layer includes complete inspectable records of agent actions for work-product and privilege review.

“complete inspectable records of agent actions for work-product and privilege purposes.”summary, stated

Tradeoffs

Own the agent runtime.

Chose A custom Harvey runtime with provider normalization and ZDR lifecycle controls over Relying only on managed agent runtimes from model labs or cloud providers.

Rationale: Runtime ownership increased engineering burden but was necessary for multi-model routing, zero retention, cost control, conflict-aware governance, and legal-specific deployment requirements.

“why Harvey built their own cloud agent infrastructure rather than relying on managed agent runtimes”summary, stated
“three critical requirements specific to serving law firms and regulated enterprises”summary, stated

Prefer true ZDR over managed state persistence.

Chose Session-scoped state inside Harvey's security boundary over Managed runtime persistence of working memory, files, tool results, and checkpoints.

Rationale: Managed persistence helps reliability, but for privileged legal data it conflicts with zero-retention requirements.

“A managed runtime earns its value precisely by persisting all of this state in its cloud”summary, stated
“Automatic state persistence and zero retention are mutually exclusive”summary, stated

Route by efficient sufficiency.

Chose The cheapest and fastest model that meets task quality over The strongest model for every legal task.

Rationale: As legal tasks become intelligence-saturated, routing everything to a frontier model overspends and prevents economical full-document-set agents.

“The goal has shifted from finding the best model to finding the one that's good enough, cheapest, and fastest for the specific task.”summary, stated
“using a top frontier model simply overspends capability the task doesn't require.”summary, stated

Build a durable legal-specific layer while absorbing ecosystem improvements.

Chose Custom legal governance, data residency, sovereign deployment, and audit capabilities over Routing around provider ecosystems indefinitely.

Rationale: Harvey expects general cloud capabilities to improve, but legal-specific requirements remain durable and need a layer Harvey controls.

“They've built their runtime to absorb improvements as they arrive rather than route around the ecosystem indefinitely.”summary, stated
“the legal-specific layer—sovereign deployments, conflict-aware governance, jurisdiction-specific data residency”summary, stated

Gotchas (what bit them)

Managed runtimes can create lock-in at the agent workforce level, not just the model level.

Root cause: Agents built inside a provider runtime depend on that provider's formats and orchestration, so switching providers means moving the operational agent system too.

Resolution: Harvey built a provider abstraction layer that normalizes harnesses, sandboxes, formats, streaming behavior, and failure modes.

3 supporting quotes
“the lock-in extends beyond just the model to the entire agent workforce”summary, stated
“live inside that provider's runtime using its formats and orchestration.”summary, stated
“built an abstraction layer that normalizes the different agent harnesses, sandboxes, and behavioral differences”summary, stated

Zero data retention cannot be added as a deletion endpoint after a run.

Root cause: Deleting persisted state after execution still means data was retained, which violates ZDR requirements for privileged legal data.

Resolution: Harvey designed ZDR as a runtime property where data is never written to persistent storage and session state is purged afterward.

“The case study emphasizes that ZDR cannot be bolted on after the fact. There is a tempting shortcut of storing data during a run and calling a deletion endpoint afterward, but this constitutes retention followed by deletion, not zero retention.”summary, stated
“True ZDR means data is never written to persistent storage in the first place”summary, stated

Stateful agents make ZDR harder than stateless chat.

Root cause: Long-running agents need working memory, intermediate files, tool results, and checkpoints, and managed runtimes often persist that state to deliver reliability.

Resolution: Harvey keeps the whole agent lifecycle inside its boundary and scopes state to the session.

3 supporting quotes
“Agents make ZDR harder to achieve than chat applications because agents are stateful.”summary, stated
“working memory, intermediate files, tool results, and checkpoints”summary, stated
“State is scoped to the session and purged afterward, so the zero-retention guarantee covers the whole workflow rather than just the final model call.”summary, stated

Frontier-only routing can make full-corpus legal agents economically impossible.

Root cause: Large legal workflows can require hundreds of model/tool calls over enormous document sets, and many subtasks no longer need the largest model.

Resolution: Harvey routes by task quality threshold and cost, including open-source models and sandbox optimization.

3 supporting quotes
“serving capable models as agents at scale is extraordinarily expensive.”summary, stated
“the naive approach of routing everything to the best frontier model economically unsustainable.”summary, stated
“Owning the runtime allows Harvey to route each task to the most efficient model that meets the quality threshold, including open-source models they host themselves. It also enables optimization of the sandbox—how files are loaded, how work is parallelized, how compute is sized—around legal workloads”summary, stated

Provider-specific tool behavior causes cross-model quality drops.

Root cause: Each provider exposes different tool-call formats, stop conditions, streaming behavior, failure modes, and sandboxes, and tasks tuned for one model can underperform on another.

Resolution: Harvey normalizes those provider differences behind a single interface before making model choice a routing decision.

“Each provider exposes different tool-call formats, stop conditions, streaming behavior, failure modes, and execution sandboxes”summary, stated
“different tool-call formats, stop conditions, streaming behavior, failure modes, and execution sandboxes, and the same task tuned for one model will underperform on another. The abstraction layer presents a single interface above which the choice of model becomes just a routing decision. ##”summary, stated

Evaluation

Harvey evaluates models with LAB, a Legal AI Benchmark split by practice area and task type and graded with rubrics. Those benchmark results feed routing decisions so the runtime can choose the most efficient allowed model that satisfies quality thresholds as model capabilities evolve.

Metrics: LAB benchmark performance by practice area · LAB benchmark performance by task type · Quality threshold for routing · Cost per workload · Open-source versus frontier quality spread · 3-5x cost reduction versus frontier-only baseline · Provider capacity, uptime, and feature availability

3 supporting quotes
“Harvey's LAB (Legal AI Benchmark) shows clear separation in model performance by practice area and task type”summary, stated
“The benchmark-driven approach to model selection, using their LAB benchmark, represents a mature LLMOps practice.”summary, stated
“models across practice areas and task types with rubrics graded by GPT-5.4 (mentioned in the figure caption), enabling data-driven routing decisions. This suggests a continuous evaluation pipeline that feeds into their routing logic, allowing them to update routing strategies as model capabilities”summary, stated

Outcomes

The summary describes workflows across hundreds of thousands of documents, full firm document sets of hundreds of thousands or millions of files, single agent runs with hundreds of model/tool calls, 3-5x cost reductions, LAB benchmark routing, and model-rubric grading by GPT-5.4.

4 supporting quotes
“## Long Summary ## Overview Harvey is a legal AI company that has evolved from a chat-based product to a full cloud agent platform capable of completing complex legal tasks end-to-end, such as reviewing entire data rooms and producing issues lists across hundreds of thousands of documents.”summary, stated
“State is scoped to the session and purged afterward, so the zero-retention guarantee covers the whole workflow rather than just the final model call.”summary, stated
“empirically seeing 3-5x cost reductions versus a frontier-only approach”summary, stated
“This level of optimization is structurally unavailable to anyone building on top of someone else's runtime and represents the difference that makes serving agents over a firm's full document set (hundreds of thousands or millions of files) economically viable rather than a spiraling cost center.”summary, stated