Scaling AI Infrastructure for Legal AI Applications at Enterprise Scale

harvey · Legal · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise LLM inference infrastructure for legal AI

Harvey needed to run many legal AI products across multiple model providers and deployments while keeping latency, availability, quotas, observability, security, and developer access under control.

Constraints:

• Harvey processes billions of prompt tokens, hundreds of millions of output tokens, and millions of daily requests.
• Model usage is bursty and features have different request shapes and prompt-token weights.
• Model deployments have finite capacity and can degrade under concurrent load or traffic spikes.
• Developers need low-friction model access without bypassing instrumentation, security, or quotas.
• Every model call needs attribution and real-time visibility for reliability, cost, and finance analysis.

Architecture

Harvey built a centralized Python inference client around model configurations, endpoint health, weighted model selection, distributed quota/rate limiting, a Redis-backed token-bucket algorithm, an OpenAI-compatible model proxy, telemetry, Snowflake accounting, and strict SLA burn-rate alerts. The library abstracts model calls for products and developers while routing traffic across healthy deployments and preventing local experiments or bursty features from degrading shared model capacity.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Centralized Python client"] --> B["Model configurations"]
  A --> C["Model endpoint selector"]
  C --> D["Model health tracker"]
  C --> E["Model deployments"]
  A --> F["Quota and rate limiter"]
  F --> G["Redis key-value store"]
  H["Model proxy"] --> E
  H --> I["Telemetry pipeline"]
  A --> I
  I --> J["Snowflake warehouse"]
  I --> K["SLA burn-rate alerts"]

Components

Data flow

A Harvey feature or developer request enters the centralized Python client with context such as feature, environment, user, workspace, model family, and prompt-token weight. The client chooses a healthy deployment using SLI/SLA-aware weighted selection and fallback priority, checks quota through Redis-backed distributed rate limiting, forwards external requests through the proxy when needed, emits telemetry, and exports prompt/output-token accounting to Snowflake.

Why this architecture

Harvey centralized inference because independent feature teams and developers would otherwise create untracked load, quota, security, and reliability problems. A shared client and proxy keep model onboarding fast while still enforcing endpoint health checks, fallback, rate limits, telemetry, and cost accounting across all environments.

Models used: multiple model families · model providers · model deployments

Scale

• Harvey consumes billions of prompt tokens from model providers.
  “Harvey consumes billions of prompt tokens from our model providers”article, stated
• Harvey produces hundreds of millions of output tokens.
  “produces hundreds of millions of output tokens across millions of daily requests.”article, stated
• Harvey processes millions of daily requests.
  “produces hundreds of millions of output tokens across millions of daily requests.”article, stated
• Runtime quota changes can be applied across geographically deployed clusters in seconds without restart.
  “allows runtime configuration of limits and quotas across all our geographically deployed clusters—without any restart and in just seconds.”article, stated

Alternatives considered (roads not taken)

Direct model calls from product code

Rejected because: A centralized client is needed to abstract model interactions and enforce reliability features.

Share provider API keys directly with developers

Rejected because: Direct access would bypass tracking, instrumentation, security, and resource protection.

Static rate limits requiring restart

Rejected because: On-callers need to change limits and quotas in seconds during model incidents.

Single deployment per model family

Rejected because: Parallel deployments, weighted selection, retries, and fallbacks provide higher availability under health or capacity variation.

Patterns

model-routing (model-ops)

Harvey chooses among model deployments based on model family, health, capacity, region, and fallback priority.

fallback-degradation (reliability)

Layered fallbacks and retries preserve availability when model endpoints fail health or SLA checks.

cost-aware-routing (model-ops)

Harvey evaluates request context, prompt-token weight, quotas, and capacity before sending traffic to model servers.

observability-tracing (operations)

Harvey instruments model calls, tracks token accounting, exports telemetry, and uses burn-rate SLA alerts.

privacy-preserving-processing (governance)

The proxy masks and protects sensitive data while preventing raw API-key sharing from bypassing controls.

Tradeoffs

Rate limiter design

Chose approximate sliding-window token bucket over more exact but slower accounting.

Rationale: The system balances accuracy and speed while keeping memory constant under bursty traffic.

Developer access pattern

Chose OpenAI-compatible proxy over direct provider-key sharing.

Rationale: The proxy preserves developer convenience while adding instrumentation, centralized onboarding, key rotation, and sensitive-data protection.

Infrastructure scope

Chose centralized horizontal scaling abstractions over feature-specific model integration.

Rationale: A central client lets Harvey add models quickly while keeping reliability and usage controls consistent.

Gotchas (what bit them)

Bursty model traffic can degrade shared inference servers.

Root cause: Features vary in request volume, prompt-token count, and concurrency against finite model capacity.

Resolution: Harvey uses active load balancing, weighted endpoint selection, and Redis-backed feature-aware rate limiting.

Developer experiments can unintentionally harm production deployments.

Root cause: Local machines, CI/CD, tests, and experiments can call shared model infrastructure without central accounting.

Resolution: Harvey routes external requests through a thin OpenAI-compatible proxy back into the cluster.

Endpoint health can change independently across deployments.

Root cause: Latency, success rate, capacity, and region differ across parallel model deployments.

Resolution: Harvey periodically measures SLIs and changes endpoint weights or fallback order.

Evaluation

Harvey evaluates infrastructure health through latency and success-rate SLIs, SRT/SLA thresholds, model benchmarks, statistical significance for quality and cost, request throughput/latency under burst, burn-rate alerts, and prompt/output token accounting.

Metrics: Billions of prompt tokens · Hundreds of millions of output tokens · Millions of daily requests · Latency and success-rate SLIs · SLA burn-rate alerts · Prompt and output token accounting · Runtime quota changes in seconds

Outcomes

• Harvey centralized model inference behind a Python client used by products and developers.
• Weighted endpoint selection, fallbacks, and retries preserve availability across deployments.
• Redis-backed quota/rate limiting handles bursty traffic without significant throughput or latency impact.
• The OpenAI-compatible proxy lets developers use models with instrumentation and security controls.
• Telemetry and Snowflake exports give data and finance teams token-consumption accounting.

Harvey reports billions of prompt tokens, hundreds of millions of output tokens, millions of daily requests, geographically deployed clusters, runtime quota changes in seconds, and active latency/success-rate monitoring.