Scaling Agent-Based Architecture for Legal AI Assistant

harvey · Legal · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI assistant feature scaling

Harvey needed to scale its Assistant from bespoke LLM orchestration into a shared agent framework so multiple teams could add retrieval, drafting, review, integrations, and product modes without a single team becoming the bottleneck or degrading quality.

Constraints:

Architecture

Harvey moved Assistant to a pure agent framework in mid-2025. Top-level thread interfaces became agents, formerly forced retrieval calls and bespoke editing logic became tool calls, and feature capabilities were packaged as Tool Bundles that include tools, sub-agents, and partial prompt instructions. The team adopted the OpenAI Agent SDK to avoid custom orchestration, gave feature teams controlled prompt/tool contribution boundaries, and enforced leave-one-out eval gates for Tool Bundles and system prompt upgrades.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Assistant thread agent"] --> B["OpenAI Agent SDK"]
  A --> C["System prompt"]
  A --> D["Tool Bundles"]
  D --> E["Retrieval capabilities"]
  D --> F["Drafting capabilities"]
  D --> G["Review and integration capabilities"]
  C --> H["Eval gates"]
  D --> H
  H --> I["Feature teams"]
  I --> D
  E --> A
  F --> A
  G --> A
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Assistant thread agent Top-level conversational interface that plans and executes legal tasks in one thread. OpenAI Agent SDK, System prompt, Tool Bundles
OpenAI Agent SDK External agent framework that encourages tool-calling loops over bespoke workflow orchestration. Assistant thread agent
System prompt Central reasoning instructions shared across capabilities and susceptible to cross-team conflicts. Tool Bundles, Eval gates
Tool Bundles Portable capability packages containing tools, sub-agents, and scoped instructions. Retrieval capabilities, Drafting capabilities, Review and integration capabilities, Eval gates
Retrieval capabilities Knowledge-source and tax-law retrieval tools used inside Assistant. Assistant thread agent
Drafting capabilities Drafting and editing tools/sub-agents available inside the thread. Assistant thread agent
Review and integration capabilities Document review tables, Ask LexisNexis, iManage, Deep Research, and other product integrations. Assistant thread agent
Eval gates Datasets, evaluators, thresholds, and leave-one-out tests that block regressions. Feature teams
Feature teams Teams contributing new Assistant capabilities with minimal centralized-team involvement. Tool Bundles

Data flow

A user asks a legal task in an Assistant thread. The agent reasons over the system prompt and available Tool Bundles, chooses retrieval, drafting, review, LexisNexis, iManage, or Deep Research tools as needed, executes tool calls in a loop, and combines results in-thread. Feature teams package new capabilities as Tool Bundles with instructions and datasets. Any system prompt or bundle change runs eval gates, including leave-one-out validation, before deployment.

3 supporting quotes
“The Assistant team’s goal is to proactively plan, create, and execute end‑to‑end tasks on our customers’ behalf, using tools from inside and outside of Harvey.”article, stated
“Forced retrieval calls became tool calls, new integrations became tool calls, and bespoke editing logic became (you guessed it) tool calls, along with a growing system prompt.”article, stated
“Tool Bundles and system prompt upgrades must pass leave-one-out gates to be deployed.”article, stated

Why this architecture

Harvey adopted agents because open-ended legal tasks and many integrations fit a model-driven tool-calling loop better than bespoke orchestration. Tool Bundles created a contribution interface for feature teams, while eval gates protected quality as capabilities multiplied.

3 supporting quotes
“These two problems — open‑ended problem solving and many integrations — are well‑suited to an agent framework because it can cleanly separate capabilities”article, stated
“Adopting a single agent framework scaled our in‑thread feature development from one team to four, led to emergent feature combinations, and enabled centralized eval.”article, stated
“Tool Bundles gave teams a safe contribution model, and eval gates protect quality as our capabilities multiply.”article, stated

Models used: OpenAI Agent SDK · Foundation models · Drafting sub-agent

Scale

Alternatives considered (roads not taken)

Continue bespoke Python plus LLM orchestration

Rejected because: It produced tuned systems but limited feature discoverability, retrieval integration, and cross-team contribution paths.

“Before agents, the Assistant team’s AI feature development was straightforward: write Python, mix it with LLMs, run an eval, then ship it.”article, stated

One centralized team integrates every new capability

Rejected because: Harvey needed feature and integration developers to contribute without bottlenecking on Assistant owners.

“One team shouldn’t own adding all new capabilities, or we can’t scale feature development.”article, stated

Let all teams edit the core system prompt freely

Rejected because: Prompt goals can collide, creating English merge conflicts and quality regressions.

“as soon as multiple engineers modify the system's core instructions, there's the potential to step on each other's toes.”article, stated

Write a custom agent library with workflow-style orchestration

Rejected because: The OpenAI Agent SDK forced the team to lean into model tool-calling loops instead of rebuilding hybrid orchestrators.

“We decided to adopt an external library, the OpenAI Agent SDK, instead of writing our own agent library.”article, stated

Patterns

tool-use-orchestration (orchestration)

Harvey converted retrieval, integrations, and bespoke editing logic into agent tool calls inside a shared thread interface.

“Forced retrieval calls became tool calls, new integrations became tool calls, and bespoke editing logic became (you guessed it) tool calls”article, stated
“Rather than being negative, this lack of flexibility forced our team to work with the strengths of the newest generation of models — calling tools in a loop — rather than building hybrid systems.”article, stated

multi-agent-orchestration (orchestration)

Tool Bundles can package multiple tools and sub-agents, such as a drafting bundle with an editing tool and drafting sub-agent.

“A Tool Bundle is an interface we designed that allows developers to package together new capabilities, which may be composed of multiple tools or sub-agents”article, stated
“A drafting Tool Bundle comprises an editing tool and a drafting sub-agent.”article, stated

eval-driven-development (evaluation)

Tool Bundles and prompt upgrades must pass leave-one-out gates, datasets, evaluators, and thresholds before deployment.

“3. Eval Gates on Capabilities: Tool Bundles and system prompt upgrades must pass leave-one-out gates to be deployed.”article, stated
“When any change is made to the system, developers can verify that their capability has not regressed.”article, stated

prompt-versioning (prompting)

Harvey treated system prompt and Tool Bundle instruction changes as shared artifacts that need ownership boundaries and regression gates.

“But as we started developing with agents, we realized that each new capability required its own set of directions within the main system prompt. And as soon as multiple engineers modify the system's core instructions, there's the potential to step on each other's toes.”article, stated
“> “As one of our developers put it, ‘You're no longer merging unit-testable code, you're merging English.’””article, stated

structured-output-contract (interfaces)

Tool Bundles define stable capability packages and contribution boundaries for feature teams.

“Tool Bundles give feature and integration developers the freedom to inject instructions into the main agent system prompt to achieve their capabilities”article, stated
“They also enable capabilities to be portable between different agents. Most importantly, Tool Bundles allow us to apply leave-one-out validation gating on key benchmarks.”article, stated

Tradeoffs

Agent framework versus bespoke orchestration

Chose OpenAI Agent SDK and no custom orchestration over custom workflow-style orchestrators.

Rationale: The framework forced the team to exploit tool-calling strengths of modern models and unlocked integrations across knowledge sources and Deep Research.

“Rather than being negative, this lack of flexibility forced our team to work with the strengths of the newest generation of models — calling tools in a loop”article, stated
“While we couldn't guarantee determinism in execution for the case law product, it turned out that we hit high rates of recall with a tuned prompt. And by adhering to the framework, we immediately unlocked integration with other knowledge sources and our deep research system.”article, stated

Capability contribution model

Chose Tool Bundles with partial prompt control over central Assistant-team integration for every feature.

Rationale: Tool Bundles let feature teams ship capabilities while keeping boundaries and portability.

“Tool Bundles give feature and integration developers the freedom to inject instructions into the main agent system prompt to achieve their capabilities without needing to make a request of the Assistant team. They also enable capabilities to be portable between different agents.”article, stated
“Most importantly, Tool Bundles allow us to apply leave-one-out validation gating on key benchmarks.”article, stated

Quality control

Chose leave-one-out eval gates over trusting individual feature evals or manual coordination.

Rationale: Capabilities can interfere with each other, so gates must detect when one bundle or prompt upgrade harms another.

“2. Tool Bundle → System Prompt conflicts: If a Tool Bundle mandates that the model must “always call every tool,” the agent will call tools in other bundles.”article, stated
“When any change is made to the system, developers can verify that their capability has not regressed.”article, stated

Gotchas (what bit them)

Mode-specific UX hid powerful features.

Root cause: Users had to discover when to enter Draft mode or choose knowledge-source recommendations.

Resolution: Move capabilities into one in-thread agent interface that can choose tools.

“People weren't discovering Draft mode. Integrating multiple retrieval calls behind a single interface was complex to maintain.”article, stated
“Our system integrates core building blocks — retrieval, drafting, and review — into one powerful thread. Depending on the question a customer asks, the correct answer might be:”article, stated

Agent collaboration creates prompt conflicts.

Root cause: Different feature teams can encode incompatible instructions in the shared system prompt.

Resolution: Use Tool Bundles to scope partial prompt control and eval gates to catch regressions.

“If developer A is focused on improving tool recall for retrieval tools, they might be tempted to tell the system prompt: "Call all the tools at your disposal."”article, stated
“If developer A is focused on improving tool recall for retrieval tools, they might be tempted to tell the system prompt: "Call all the tools at your disposal." However, developer B might be working on reducing the average latency”article, stated

Tool output can cause context rot.

Root cause: New tools may inject significant context that degrades agent behavior.

Resolution: Require bundle-specific datasets, evaluators, thresholds, and leave-one-out gates.

“3. Context rot: If the new tool outputs significant context, the agent can suffer from context rot.”article, stated
“Harvey guards against this by requiring the Assistant team and feature and integration developers to maintain datasets and evaluators”article, stated

Framework adoption reduced determinism.

Root cause: Moving from deterministic bespoke workflows to tool-calling loops meant execution could not be guaranteed in the same way.

Resolution: Tune prompts, accept the agent framework boundary, and rely on evals to validate recall and integration quality.

“While we couldn't guarantee determinism in execution for the case law product, it turned out that we hit high rates of recall with a tuned prompt.”article, stated

Evaluation

Harvey evaluates agent changes with centralized evals, Tool Bundle datasets, evaluators, thresholds, leave-one-out gates, and capability-specific recall checks.

Metrics: In-thread feature development scaled from one team to four · Tool Bundles and system prompt upgrades must pass leave-one-out gates · Retrieval datasets define many queries with expected recall across knowledge sources · Future work includes scaling eval gates to combinatorial Tool Bundle interactions

3 supporting quotes
“Adopting a single agent framework scaled our in‑thread feature development from one team to four, led to emergent feature combinations, and enabled centralized eval.”article, stated
“our retrieval dataset defines a large number of queries with expected recall across a set of knowledge sources.”article, stated
“Scaling our eval gating framework to more capabilities — how do we smartly test for a combinatorial number of Tool Bundles?”article, stated

Outcomes

The case centers on organizational and capability scale: one to four teams, multiple integrations, top-level thread agents, and leave-one-out gates for every Tool Bundle or system prompt upgrade.

“Agents allow you to scale feature development in thread products to the size of your engineering team, but they are a deceptively complex surface to develop.”article, stated
“this shift to an agent framework ultimately enabled us to scale feature development and accelerate delivery.”article, stated