Scaling AI Agents in Production for B2B Growth and Outreach

clay · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: B2B growth and outbound automation

Clay needed to scale AI agents for finding, closing, and growing customers from early chat-completion wrappers to hundreds of millions of monthly agent runs while preserving quality, throughput, cost visibility, model flexibility, and debuggability across customer-specific workflows.

Constraints:

• Clay agents run at very large volume: about 300 million agent runs per month.
• Individual agent runs can take 10 to 30 steps across web research, crawling, synthesis, and content generation.
• Users provide broad natural-language inputs, so production behavior varies widely by customer and use case.
• The platform must reconcile cost across inference providers accurately enough for finance and engineering.
• Model preferences and model capabilities vary by user, provider, and workflow.

Architecture

Clay evolved into a software-plus-agent platform with agents for prospect search, account reasoning, outbound messaging, lead enrichment, routing, and customer growth. The production stack uses an internal agent harness on the Vercel AI SDK, LangSmith for traces, evaluations, production profiling, cost reconciliation, and debugging, a metaprompter for model routing, and customer/engineering/support access to traces for diagnosing quality and cost issues.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Clay users"] --> B["Clay agent products"]
  B --> C["Agent harness"]
  C --> D["Web and database research"]
  C --> E["Metaprompter"]
  C --> F["LangSmith"]
  B --> G["Customer workflows"]
  F --> H["Engineering and support teams"]
  H --> B
  G --> A

Components

Data flow

Customers enter natural-language research or go-to-market tasks. Clay agents search the web and Clay's company/person database, crawl pages, synthesize documents, produce qualification or outbound recommendations, and send results into sequencers, lead routing, enrichment, or CRM workflows. LangSmith receives traces from the agent harness, supports offline and online evaluation, exposes production traffic patterns, and exports data for analysis and cost reconciliation.

Why this architecture

Clay's scale makes manual inspection impossible, so trace visibility, offline evaluation, online profiling, and cost reconciliation became core platform capabilities. The agent harness remains model-agnostic because users prefer different providers and model profiles, and the metaprompter uses evaluations to map prompts to suitable models.

Models used: OpenAI models · Anthropic models · LLMs · LLM-as-judge

Scale

• Clay runs around 300 million agent runs per month.
  “today we're running around 300 million agent runs a month.”transcript, stated
• Clay agent runs average roughly 10 to 30 steps.
  “Somewhere between, I would say 10 to 30 steps on average”transcript, stated
• Clay achieved about 99 to 99.5% cost reconciliation between LangSmith data and provider bills.
  “With LangSmith, we now have, I think it's around like a 99 maybe like 99.5% reconciliation rate between the LangSmith data and our actual bills”transcript, stated
• About 25 to 50 people at Clay have LangSmith access.
  “I'd say it's probably on the order of 25 to 50.”transcript, stated

Alternatives considered (roads not taken)

Manual inspection of agent behavior at scale

Rejected because: Once agent runs reach hundreds of thousands or millions, manual inspection and customer-by-customer conversations break down.

One fixed model provider

Rejected because: Different users prefer different providers and models behave differently by writing style and task profile.

Build observability entirely in-house

Rejected because: LangSmith's agent-framework orientation, nested trace support, and fast integration fit Clay's needs.

Patterns

observability-tracing (observability)

Clay exports traces from its agent harness to LangSmith from day zero, using them for production profiling, nested subagent visibility, debugging, and support investigations.

eval-driven-development (evaluation)

Clay uses offline evaluations, ground-truth comparisons, online evaluators, and production trace analysis to benchmark quality against cost and throughput.

llm-as-judge (evaluation)

Clay uses LLM-as-judge for research-and-synthesis outputs where deterministic checks are insufficient.

model-routing (model-ops)

Clay maps prompts to model providers through a metaprompter based on workflow profile and evaluation results.

tool-use-orchestration (orchestration)

Clay agents combine web search, crawling, database search, synthesis, recommendations, outbound messaging, lead enrichment, and routing in multi-step workflows.

Tradeoffs

Model strategy

Chose model-agnostic routing and user model choice over one default provider for all workflows.

Rationale: Different models have different writing styles and task profiles, and users develop provider preferences.

Observability platform

Chose LangSmith over Clay-only in-house tracing and evaluation tooling.

Rationale: LangSmith understood nested agent traces, integrated quickly with the Vercel AI SDK, and supported both production profiling and zero-to-one agent building.

Evaluation mode

Chose a mix of deterministic matches and LLM-as-judge over only deterministic ground-truth matching.

Rationale: Some agent outputs are factual and structured, while research synthesis and recommendations need judgment-based scoring.

Gotchas (what bit them)

At high run volume, ground truth becomes unknowable without observability.

Root cause: Customers use blank text boxes for broad tasks, creating too many behaviors to inspect manually.

Resolution: Clay uses production profiling, online evaluators, and trace analysis to understand behavior at scale.

Customer reports do not identify the failing layer by themselves.

Root cause: A bad answer can come from a tool problem, an inference problem, or another layer in the stack.

Resolution: Clay uses LangSmith traces to inspect where the failure occurred.

Inference cost accounting can drift from provider bills.

Root cause: Multiple inference providers and high run volume made spending opaque before LangSmith reconciliation.

Resolution: Clay reconciles LangSmith data against provider bills at roughly 99 to 99.5% coverage.

Evaluation

Clay evaluates agents through deterministic ground-truth checks, LLM-as-judge for synthesized outputs, offline quality benchmarks against cost and throughput, online evaluators for production behavior, and LangSmith traces for debugging and cost reconciliation.

Metrics: Correct data quality · Throughput · Cost · Ground-truth factual matches · LLM-as-judge scores for recommendations · Latency and errors · 99 to 99.5% provider-bill reconciliation

Outcomes

• Clay reports about 300 million agent runs per month.
• Agent runs average about 10 to 30 steps across research, crawling, synthesis, and outbound workflows.
• LangSmith gives Clay production profiling, trace debugging, offline evaluation, and cost reconciliation.
• Cost reconciliation reached roughly 99 to 99.5% between LangSmith data and provider bills.
• Support teams use LangSmith to debug customer traces.
• Clay is exploring trace-derived skills and memory updates for longer-horizon agents.

Clay operates about 300 million agent runs per month, with 10 to 30 steps per run, 25 to 50 LangSmith users, and 99 to 99.5% cost reconciliation coverage.