Building Production-Ready Customer Support AI Agents: Challenges and Solutions

gradient_labs · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise customer-support AI agents

Gradient Labs needed to run AI customer-support agents in production for enterprises where the agent must handle live conversations, race conditions, knowledge quality, private tool calls, model/provider constraints, enterprise policies, and safe pre-deployment testing.

Constraints:

• Customer support conversations change while the agent is still working, creating fast/slow race conditions with customers and human agents.
• LLM calls can fail successfully: the API request completes, but the completion may still be bad.
• Support requests differ in urgency, from time-bound customer replies to delay-tolerant knowledge validation and documentation processing.
• Knowledge bases built by simply vectorizing internal docs produce bad outcomes because docs are incomplete, private, or not written for agents.
• Enterprise tools may expose private or sensitive APIs that cannot be called freely during evaluation.
• Enterprise customers impose language, handoff, channel, timing, and intent restrictions that must be configurable.

Architecture

Gradient Labs models its production support agent as a state machine. Conversation events from support platforms trigger state transitions; some states are deterministic and others invoke agentic behavior. The agentic layer keeps code at a behavioral level, returns events for observability, executes all LLM calls through Temporal for durable retries, abstracts model providers such as Bedrock, Azure, and direct providers, logs completions and costs, uses point-in-time knowledge references for debuggability, and tests PR versions through thousands of simulations on production-grade traffic in an isolated evaluation environment.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer[Customer] --> support-platform[Support platform]
  support-platform --> agent-state-machine[Agent state machine]
  agent-state-machine --> agentic-behavior-layer[Agentic behavior layer]
  agent-state-machine --> enterprise-policy-controls[Enterprise policy controls]
  enterprise-policy-controls --> human-support-agent[Human support agent]
  support-platform --> human-support-agent
  agentic-behavior-layer --> knowledge-layer[Knowledge layer]
  knowledge-layer --> point-in-time-knowledge-store[Point-in-time knowledge store]
  agentic-behavior-layer --> tool-invocation-layer[Tool invocation layer]
  tool-invocation-layer --> private-company-apis[Private company APIs]
  agentic-behavior-layer --> llm-execution-layer[LLM execution layer]
  llm-execution-layer --> temporal-durable-execution[Temporal durable execution]
  llm-execution-layer --> model-provider-abstraction[Model provider abstraction]
  agentic-behavior-layer --> event-log[Event log]
  llm-execution-layer --> event-log
  tool-invocation-layer --> simulation-environment[Simulation environment]
  event-log --> simulation-environment
  simulation-environment --> evaluation-reviewers[Evaluation reviewers]

Components

Data flow

A customer starts or updates a support conversation. The support platform emits events into the agent state machine, which decides whether to respond, stop prior work, wait, escalate, or move into an agentic behavior. The behavior layer retrieves point-in-time knowledge, invokes private tools when safe, makes LLM calls through Temporal and the provider abstraction, returns events for observability, and sends a response or handoff. Before deployment, a PR version runs thousands of simulations on production-grade traffic in an isolated evaluation environment.

Why this architecture

Gradient Labs uses state machines and durable execution because production support agents are slower and more stateful than traditional APIs, must stop or change work when external conversation state changes, and have many more failure modes than a traditional model-serving system. State machines expose high-level flow control; the agentic middle layer stays readable; Temporal and provider abstraction hide low-level retries and infrastructure details from AI engineers.

Models used: LLMs · OpenAI · Anthropic · Google · Bedrock · Azure · Temporal

Scale

• Gradient Labs was live with a few companies a few months after launch.
  “We've launched ours um a few months ago and we're now running it live with a few companies”transcript, stated
• A basic agentic workflow may make two or three LLM calls and be much slower than a database API call.
  “even a basic AI agentic workflow that's going to make two or three llm calls is going to be much much much slower than uh you know an API call”transcript, stated
• RAG alone would address only around 10% or less of the largest enterprises' support problems.
  “just a rag based agent is going to probably only cut into 10% or less of the the the actual problems of what their humans are working on”transcript, stated
• The agent was processing hundreds of conversations per day.
  “we are live uh with a few Partners we've been running hundreds of of conversations per day”transcript, stated
• PR-based evaluation runs thousands of simulations.
  “spin up a PR of a version of our agent run you know thousands of simulations”transcript, stated
• One NFR example assumes the AI handles roughly 70-80% of support and hands off the remaining 20%.
  “if you imagine an AI agent that is only going to take off uh 70 or 80% of your customer support then for that rest of the 20%”transcript, stated

Alternatives considered (roads not taken)

Treat the support bot as simple RAG

Rejected because: For large enterprises, pure RAG covers only a small fraction of what human support agents handle.

Dump all documentation into a vector database

Rejected because: Internal docs are not written for agents, omit knowledge in people's heads, mix public/private content, and approximate retrieval is insufficient for high-quality support.

Adopt an LLM framework immediately

Rejected because: Gradient Labs wanted deep surgery and a developer experience at the behavioral layer before committing to a framework.

Use traditional A/B rollouts or shadow deployments as the main agent test path

Rejected because: AI agent testing is multifaceted, so Gradient Labs runs isolated production-grade simulations before live deployment.

Let the agent freely call private tools during evaluation

Rejected because: Enterprise APIs can be private or sensitive, so evaluation has to test tool-using behavior without unrestricted real calls.

Bind prompts cleanly away from specific models

Rejected because: The speaker argues prompt performance is strongly tied to the model invoked, so abstraction has limits.

Patterns

durable-execution (orchestration)

Gradient Labs runs all LLM calls through Temporal so AI engineers can focus on behavior instead of retries and low-level execution failures.

async-message-orchestration (orchestration)

The support agent is modeled as a state machine that reacts to conversation starts, new customer messages, missing signals, timers, and external state changes.

observability-tracing (production-operations)

Every agent decision returns events, and completions and costs are logged so the team can observe production behavior and simulation outcomes.

model-routing (model-operations)

The system abstracts providers so engineers can choose models without caring whether they come from Bedrock, Azure, or another source.

rag-retrieval (knowledge)

Gradient Labs uses knowledge retrieval but treats naive RAG as insufficient, emphasizing cleaned, correct, point-in-time knowledge for customer support.

eval-driven-development (quality)

Gradient Labs evaluates PR versions through thousands of production-grade simulations and manual or automatic review before live deployment.

human-in-the-loop-gate (workflow)

The support framework must know when to hand off to humans and constrain the AI to languages or intents that human agents can support.

guardrails-io-filtering (policy)

Enterprise policy controls restrict languages, conversation closure, timing, channels, and sensitive intents such as fraud.

Tradeoffs

Model agent flow as a state machine

Chose State-machine orchestration with deterministic and agentic states over Ad hoc prompt/tool chains.

Rationale: State machines make external inputs, timers, signal changes, and conversation control explicit while keeping high-level flow separate from low-level implementation.

Keep behavior code between states and raw LLM calls

Chose A readable behavioral layer over Coding directly at state-machine or LLM-call level.

Rationale: Engineers can express what the agent is doing without drowning in orchestration details or raw model-call plumbing.

Use durable execution for LLM calls

Chose Temporal-backed LLM execution over Manual retry and failure handling in agent code.

Rationale: Durable execution abstracts low-level retry/failure mechanics away from AI engineers.

Abstract providers but preserve model choice

Chose Model/provider abstraction across Bedrock, Azure, and other providers over Provider-specific implementation details in agent behavior code.

Rationale: The agent can use different models across providers while engineers request the model they want without caring about source plumbing.

Test directly against production-grade traffic in isolation

Chose PR agent versions in an isolated evaluation environment over Traditional local-dev to staging to live-production path.

Rationale: Production-grade traffic exposes the weird edge cases agents need, while isolation avoids affecting real customers.

Restrict agent behavior by enterprise policy

Chose Functional and behavioral controls for languages, timers, channels, and intents over A universally capable support agent.

Rationale: Enterprise handoff capacity and regulated-topic constraints require the agent to know what it may say and when to escalate.

Gotchas (what bit them)

A new customer message can invalidate work already in progress.

Root cause: Agentic workflows are slower than ordinary API calls, so the outside conversation can change while the agent is still reasoning.

Resolution: Model the interface layer as stateful flow control that can stop work, restart work, or react to missing signals.

An LLM call can fail successfully.

Root cause: The API call can return normally while the completion is low quality or unsuitable.

Resolution: Treat completion quality as a separate failure class from transport/API success.

Naive vector-database RAG produces poor support outcomes.

Root cause: Internal documents are incomplete, not agent-oriented, and may not distinguish public from private information.

Resolution: Treat knowledge engineering as a production layer with clean, correct, point-in-time references and approaches for missing tacit knowledge.

Approximate retrieval is not enough for high-quality enterprise support.

Root cause: Approximate nearest-neighbor retrieval can return insufficient results when quality matters most.

Resolution: Avoid treating ANN scale optimizations as a substitute for exact, high-quality knowledge access in critical support contexts.

Live-only knowledge prevents historical debugging.

Root cause: If the knowledge base changes after an interaction, teams cannot reconstruct what the agent knew days earlier.

Resolution: Use point-in-time knowledge references similar to feature-store time travel.

Enterprise tool data can change between retrieval and use.

Root cause: Private business systems are live state, not static public-web facts; a fraud status can change after a customer says something else.

Resolution: Reason about returned data validity and state changes when invoking tools inside a company.

Evaluation

Gradient Labs evaluates agents by spinning up PR versions in isolated production-grade environments, running thousands of simulations over live traffic scenarios, and manually or automatically evaluating outcomes. The evaluation must account for race conditions, private-tool constraints, knowledge versions, model/prompt coupling, language handoff, regulated intents, completions, and cost.

Metrics: Simulation outcomes · Manual simulation review · Automated simulation review · Conversation resolution behavior · Customer reply race conditions · LLM completion quality · Provider rate limits · Tool-call validity · Knowledge point-in-time reproducibility · Completion cost · Language handoff compliance · Intent restriction compliance

Outcomes

• Gradient Labs launched its AI support agent and ran it live with a few enterprise customers.
• The agent processed hundreds of conversations per day.
• Customers sometimes thanked the AI agent for its service.
• The production architecture uses state machines with deterministic and agentic states.
• LLM calls run through Temporal for durable execution.
• Every decision returns events for observability.
• Provider abstraction lets engineers select models without provider plumbing.
• The team concluded simple RAG would cover only around 10% or less of the largest enterprise support workload.

The source reports a few live partners, hundreds of conversations per day, thousands of PR simulations, two or three LLM calls per basic workflow, 70-80% AI-handled support as an example, and a 10% or less estimate for pure-RAG coverage in large enterprises.