Scaling Go-to-Market AI Agents at Production Scale

clay · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: go-to-market agent infrastructure

Clay needed to run hundreds of millions of go-to-market agents reliably, cheaply, and with enough quality to support outbound strategy at whole-market scale.

Constraints:

• Clay runs over 350 million go-to-market agents per month over tens of millions of companies and hundreds of millions of contacts.
• Agents spend much of their time waiting on browsers, APIs, and inference, making wall-time billing expensive.
• Throughput, cost, and quality all matter because agents process trillions of tokens weekly.

Architecture

Clay operates Claygent over proprietary company/contact datasets and web data. They moved from Lambda to ECS, then added durable workflow execution patterns with queues and checkpointing, adaptive backpressure/rate limiting similar to TCP/IP congestion control, customer fairness controls, prompt-caching-aware agent harness design, bounded retries/tool calls, and offline/online evals through tooling such as LangSmith.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Customer GTM lists"] --> C["Claygent agent harness"]
  B["Clay proprietary datasets"] --> C
  C --> D["Durable workflow layer"]
  D --> E["ECS compute"]
  C --> F["Inference throttling layer"]
  F --> G["Inference providers"]
  C --> H["Caching and retry policy"]
  H --> G
  C --> I["Quality evaluation loop"]

Components

Data flow

Customer lists and Clay's proprietary datasets feed Claygent research and scoring workflows. Agent executions run on ECS with queue/checkpoint durability, throttle against downstream inference providers with backpressure, use caching strategies and bounded retry/tool-call policies to control cost, and are evaluated offline and online against go-to-market outcomes.

Why this architecture

At Clay's scale, agent infrastructure has to optimize waiting-time economics, recovery, rate limits, cacheability, fairness, and quality together; raw model calls are only one part of the system.

Models used: Claygent · Anthropic models · LLMs via downstream inference providers · LangSmith evaluation tooling

Scale

• Clay runs over 350 million go-to-market agents every month.
  “we run over 350 million go-to-market agents every month.”transcript, stated
• Clay has a proprietary dataset of over 40 million companies and 900 million contacts.
  “We have a proprietary data set of over 40 million companies and 900 million contacts that our agents research over.”transcript, stated
• Claygent processes trillions of tokens every week.
  “It processes trillions of tokens every week.”transcript, stated
• Adaptive throttling produced 4-10x throughput versus a naive system.
  “this can yield four to ten times as much throughput as a more naive system.”transcript, stated
• Prompt caching can yield up to 70% cost savings with providers like Anthropic.
  “For providers like Anthropic, this can yield up to 70% cost savings — quite high.”transcript, stated

Alternatives considered (roads not taken)

Lambda for Claygent

Rejected because: Lambda wall-time billing was prohibitively expensive for agents that spend much of execution waiting.

Naive fixed-rate inference sending

Rejected because: Adaptive throttling against rate limits produced 4-10x more throughput.

Unbounded agent retries and research loops

Rejected because: Retries and tool calls can sprawl and inflate cost unless bounded in conjunction with evals.

Patterns

durable-execution (agent-infrastructure)

Clay uses queue/checkpoint patterns so ECS-hosted agents can recover from random host failures.

latency-optimization (throughput)

Adaptive throttling maximizes inference throughput while backing off on rate-limit pressure.

semantic-caching (cost)

Clay designs its agent harness around provider prompt-caching strategies to reduce token costs.

eval-driven-development (evaluation)

Clay uses offline and online evals to tune its go-to-market agent harness to real product tasks.

Tradeoffs

Compute platform

Chose ECS with durable workflow patterns over Lambda wall-time execution.

Rationale: Lambda was too expensive for waiting-heavy agents, but ECS required explicit reliability architecture.

Throughput control

Chose adaptive throttling/backpressure over naive traffic sending.

Rationale: Backpressure maximizes downstream inference capacity without excessive rate-limit failures.

Quality strategy

Chose offline and online evals tied to outcomes over cost/throughput optimization alone.

Rationale: Clay treats quality as the reason throughput and cost improvements matter.

Gotchas (what bit them)

Cheaper ECS compute introduced reliability work.

Root cause: Moving from Lambda reduced wall-time cost but exposed host-failure recovery needs.

Resolution: Add durable workflow execution with queues and checkpointing.

One large customer can crowd out smaller customers.

Root cause: Some customers run millions of agents over their market while new customers may run only a handful.

Resolution: Build fairness controls across customers.

Agent cost can sprawl through retries and tool calls.

Root cause: Trillions of weekly tokens make even modest inefficiencies expensive.

Resolution: Bound retries and tool calls with eval-backed stopping policies.

Evaluation

Clay evaluates agent quality with offline and online evals, product outcomes, context/data quality, and cost tied to quality and outcomes.

Metrics: agent executions · weekly tokens · throughput multiplier · prompt caching savings · offline evals · online evals · quality/outcome metrics

Outcomes

• Clay runs more than 350 million go-to-market agents monthly.
• Claygent processes trillions of tokens weekly.
• Adaptive throttling yields 4-10x throughput versus naive systems.
• Prompt caching can yield up to 70% cost savings with Anthropic.
• Clay added durable workflows, fairness controls, bounded retries, and online/offline evals.

The source is a production-scale technical talk and provides concrete throughput/cost lessons.