Building a Production AI Agent System for Customer Support

decagon · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise customer support automation

Decagon wanted enterprise support agents that could answer across channels, reason over unstructured support knowledge, execute workflows and actions, route to humans when needed, and improve through admin and QA feedback without damaging brand trust or compliance posture.

Constraints:

• support requests arrive through chat, email, SMS, voice, and other channels with different latency expectations
• customer support knowledge is often unstructured across knowledge bases, help articles, SOPs, and agent conversations
• actions such as refunds, credits, or reshipping orders require guardrails and customer-specific criteria
• some interactions must always route to humans because of risk, missing workflows, or industry constraints
• production agents need both pre-deployment test coverage and continuous monitoring as real user distributions drift

Architecture

Decagon built an AI agent engine around five connected pieces: a core AI agent brain loaded with enterprise knowledge, actions, workflows, and tools; routing to decide whether conversations stay with AI or go to humans; agent assist as a copilot for human support agents; an admin dashboard for configuration, rollout, metrics, and monitoring; and a QA interface for granular conversation review and continuous improvement.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-channels[Customer channels] --> core-ai-agent-brain[Core AI agent brain]
  customer-channels --> routing-product[Routing product]
  core-ai-agent-brain --> knowledge-sources[Knowledge sources]
  core-ai-agent-brain --> action-and-tool-library[Action and tool library]
  action-and-tool-library --> guardrail-layer[Guardrail layer]
  core-ai-agent-brain --> workflow-library[Workflow library]
  core-ai-agent-brain --> routing-product
  routing-product --> human-support-agent[Human support agent]
  core-ai-agent-brain --> agent-assist[Agent assist]
  agent-assist --> human-support-agent
  core-ai-agent-brain --> admin-dashboard[Admin dashboard]
  admin-dashboard --> qa-interface[QA interface]
  qa-interface --> human-support-agent
  admin-dashboard --> guardrail-layer

Components

Data flow

A customer contacts support through chat, email, SMS, voice, or another channel. The core agent brain uses company knowledge, workflows, and tool/action permissions to respond or act. Routing decides whether to keep the case with AI, hand it to a human, or let a human use agent assist. Conversation data from AI and human handling flows back into the admin dashboard, where customers configure knowledge, workflows, brand behavior, tests, rollout, metrics, and QA taxonomies for ongoing improvement.

Why this architecture

Decagon uses a full agent-engine loop because enterprise support needs more than answer generation. A reusable brain centralizes knowledge, workflows, and actions; channel adapters handle different timing expectations; routing and agent assist keep humans in the loop; the admin dashboard gives customers brand/configuration control; and QA/test loops catch distribution drift and edge cases after launch.

Models used: LLMs · AI agents · evaluation agent · voice model/API

Scale

• The agent brain can serve chat, email, SMS, voice, and other support channels.
  “that includes chat email SMS uh voice which is when you're calling the support number um and other channels as well”transcript, stated
• Chat users may expect responses in less than a minute, while voice requires nearly instant responses.
  “email Emil you might expect a chat response back within less than a minute but when you're sending an email you're not refreshing your email every minute expecting a response back and similarly for voice um when you're on the phone”transcript, stated

  “when you're on the phone with someone and you're having a conversation you probably expect them to reply nearly instantly”transcript, stated
• Offline test sets often start with a couple hundred conversations per workflow.
  “know maybe couple hundred conversations per um like workflow to be able to cover the different paths um but it really depends based on like the how much customer data that they have um how many workflows they are building out the kind of configuration and combination of all”transcript, stated
• Early production QA can start with rollout to 5% of users before broader launch.
  “that might look like is you roll out your AI agent to 5% of users you gather the conversation data and you have the QA team or uh the decagon team go through and um analyze these conversations so create certain you know create a QA taxonomy like maybe you're”transcript, stated
• Action criteria can include rules such as whether a user requested a refund in the last seven days.
  “if they haven't requested a refund in the last seven days and there's all these different sort of dynamic criteria that we can work with our customers to Define very quickly um and there's also criteria that is more fuzzy for”transcript, stated

Alternatives considered (roads not taken)

Use hard-coded decision trees for support

Rejected because: Customers who do not match a predefined happy path fall through and have a poor experience.

Build only a simple ReAct loop

Rejected because: Decagon uses an ecosystem of agents that can perform extra steps, review each other's work, and abstract that complexity into a unified brain.

Let AI handle all escalated interactions after handoff

Rejected because: Customers choose whether human handoff is final or whether the AI can re-enter after a configurable threshold.

Rely only on pre-deployment tests

Rejected because: Real support traffic changes over time and contains unexpected questions, requiring continuous test creation and QA.

Patterns

multi-agent-orchestration (agent-engine)

Decagon abstracts multiple cooperating agents that can do additional steps, review each other's work, and look for specific qualities into one enterprise support brain.

tool-use-orchestration (actions)

The agent brain gets a library of tools/actions such as refunds, order status lookup, credits, and reshipping, with customer-defined execution criteria.

human-in-the-loop-gate (escalation)

Routing and agent assist move cases to humans when risk, missing workflows, or customer policy requires review and approval before messages or actions are issued.

safety-policy-enforcement (guardrails)

Decagon builds guardrails for prompt injection, manipulation, compliance requirements, dangerous tools, and enterprise security around executed actions.

eval-driven-development (quality)

Decagon uses pre-deployment conversation test sets, evaluation criteria, score-returning evaluation agents, online metrics, and QA taxonomies for continuous improvement.

canary-rollout (deployment)

The QA process includes rolling the agent to a small percentage of users, reviewing conversations, and iterating before broader deployment.

Tradeoffs

Centralize enterprise support logic in one agent brain

Chose A reusable core agent brain across channels over Separate bespoke bots for every support channel.

Rationale: The same workflows and tools can serve chat, email, SMS, and voice, with channel-specific presentation and latency adaptations.

Route some cases to humans

Chose Dynamic routing plus agent assist over Forcing every support interaction through full automation.

Rationale: High-risk industries, missing workflows, and customer adoption appetite require configurable human review paths.

Customize brand behavior per enterprise

Chose Customer-specific tone, brand, structure, and workflow configuration over A generic support-agent personality.

Rationale: The agent represents the customer's brand and can damage reputation if it speaks or acts incorrectly.

Combine quantitative metrics with qualitative QA

Chose Deflection/CSAT metrics plus conversation review and taxonomies over Only tracking aggregate automation metrics.

Rationale: Aggregate metrics show performance, but granular QA reveals edge cases and concrete changes to make.

Gotchas (what bit them)

Human SOPs do not translate directly into tool-calling instructions.

Root cause: Human manuals describe behavior differently than function/tool calling specifications.

Resolution: Build explicit tool/action libraries and AI-specific workflow instructions.

Voice, chat, and email cannot share the same product latency assumptions.

Root cause: Users expect different response timing and presentation by channel.

Resolution: Reuse core workflow logic but adapt presentation, timing, and modality-specific behavior.

Dangerous support actions need customer-specific policy gates.

Root cause: Refunds, credits, and reshipping can carry financial, compliance, or abuse risk.

Resolution: Define dynamic criteria and guardrails case by case with each customer.

Offline tests will not cover future production distribution.

Root cause: Customers ask new questions as products change and users interact in unexpected ways.

Resolution: Continuously inspect real conversations, add new tests, and update QA taxonomies after launch.

Evaluation

Decagon evaluates agents before deployment with customer-built test sets and evaluation criteria, monitors online metrics such as deflection and customer satisfaction, and runs QA loops on production conversations to create new tests, labels, taxonomies, and improvement actions.

Metrics: deflection or automation rate · customer satisfaction score or NPS · tone and formatting correctness · correct information retrieval · correct workflow and tool execution · QA taxonomy labels and qualitative comments

Outcomes

• Decagon packaged customer-support automation as a full support ecosystem, not only an answering bot.
• The core agent brain can be reused across chat, email, SMS, voice, and other channels.
• Enterprises can configure brand behavior, knowledge, workflows, actions, rollout, monitoring, and QA in the admin dashboard.
• The system supports human routing and agent assist for high-risk or unsupported cases.

The source emphasizes multi-channel enterprise deployment, less-than-a-minute chat expectations, near-instant voice expectations, predeployment tests with hundreds of conversations per workflow, and 5% rollout QA loops.