Building Production AI Customer Support Agents with Multi-Agent Architecture and Human-in-the-Loop Design

lorikeet · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI customer support automation for regulated and high-volume support teams

Lorikeet needed to automate customer support work that goes beyond FAQ answering: agents had to operate inside existing ticketing systems, look up information across tools, take actions, know when to escalate, and help non-technical support teams configure and improve the system safely.

Constraints:

• Support work often requires cross-referencing multiple internal systems rather than simply retrieving knowledge-base text.
• Customers already live in ticketing platforms such as Zendesk and Intercom, so replacing those tools would create friction.
• AI support agents must be helpful without overpromising, especially in regulated domains such as fintech and healthcare.
• Many customers have incomplete, outdated, or tacit SOPs, so configuration cannot assume perfect process documentation.
• Support teams need to review, evaluate, and improve AI behavior at scale without becoming AI prompt engineers.
• Voice and live-chat channels add latency and real-time interaction pressure beyond email support.

Architecture

Lorikeet uses a two-agent architecture. The customer-facing concierge agent sits on top of existing ticketing platforms as another team member, reads instructions and business context, uses configured tools, follows guardrails, and handles customer tickets across channels. The internal coach agent helps support teams configure workflows, create tests, diagnose trace failures, write or adjust instructions, and propose guardrails. A resolution-in-the-loop pattern lets the concierge keep ticket ownership while asking a human for the one missing fact or decision needed to continue. Guardrails run as isolated LLM-as-judge checks before responses reach customers, and customer-defined eval scenarios drive SOP and guardrail improvements.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-ticket["Customer ticket"] --> ticketing-platform["Ticketing platform"]
  ticketing-platform --> concierge-agent["Concierge agent"]
  business-instructions-and-sops["Business instructions and SOPs"] --> concierge-agent
  configured-tools["Configured tools"] --> concierge-agent
  concierge-agent --> guardrail-checks["Guardrail checks"]
  business-context["Business context"] --> guardrail-checks
  guardrail-checks --> human-support-agent["Human support agent"]
  concierge-agent --> resolution-in-the-loop-request["Resolution-in-the-loop request"]
  resolution-in-the-loop-request --> human-support-agent
  resolution-in-the-loop-request --> knowledge-base["Knowledge base"]
  human-support-agent --> coach-agent["Coach agent"]
  coach-agent --> evaluation-scenarios["Evaluation scenarios"]
  coach-agent --> trace-diagnostics["Trace diagnostics"]
  coach-agent --> business-instructions-and-sops
  hybrid-configuration-ui["Hybrid configuration UI"] --> coach-agent
  evaluation-scenarios --> concierge-agent

Components

Data flow

A customer message enters an existing ticketing platform. The concierge agent acts like another support user in that platform, applies customer-specific instructions, retrieves or uses configured tools, and proposes or sends a response. Before the response reaches the end customer, isolated guardrail checks evaluate whether to steer, escalate, or allow it. If the agent is uncertain or blocked on missing knowledge, it escalates or uses resolution-in-the-loop to ask a human a narrow question, then continues the ticket. Separately, support teams use coach through a conversational and UI-hybrid interface to define good/bad examples, create simulations, inspect traces, diagnose failures, update instructions, and create guardrails.

Why this architecture

Lorikeet keeps the concierge inside existing support tooling because ticketing platforms already solve queues, triage, and human-agent workflows. It separates concierge from coach because customer-facing automation needs locked-down guardrails, while internal configuration benefits from a cooperative coding-agent-like assistant. Resolution-in-the-loop avoids the two bad extremes of either full automation or full ticket handoff, and test-case-first configuration handles messy customer SOPs better than asking non-technical users to author prompts or workflows from scratch.

Models used: LLM concierge agent · LLM coach agent · LLM-as-judge guardrail checks · large language models

Scale

• Lorikeet had been operating for about two and a half years at the time of the talk.
  “So this is two and a half odd years ago. We had this intuition that there was an overlooked unsexy part of the tech sector.”transcript, stated
• The early product processed real healthcare customer emails via CSV, CLI, spreadsheets, email, and Slack before later self-service tooling.
  “they would send us emails in CSV and we process it through our little agent through a command line interface. We'd send them the results back”transcript, stated
• Email was chosen first because sub-hour replies were considered excellent, making it a lower-latency channel for building action-taking plumbing.
  “an email reply in like less than an hour is considered excellent or was considered excellent in the days of SEO and support teams.”transcript, stated
• At mature deployments, human average handle time increased because humans spend more time on genuinely hard tickets.
  “average handle time of their human operators has actually gone up. So they're spending more time on customer support tickets.”transcript, stated
• The system uses escalation analytics such as 50% of handoffs being about one question type to prioritize knowledge-base improvement.
  “we can say 50% of the things being handed off to humans is about this kind of question. Maybe you need to improve your knowledge based article.”transcript, stated
• One prior UX change in a related ML-review workflow reportedly made users four times more efficient without changing the model.
  “all we rebuilt was the UX just the UX and yet our users were four times more efficient with their tasks”transcript, stated

Alternatives considered (roads not taken)

Build a simple knowledge-base question-answering bot

Rejected because: The team observed that support work involved multiple systems, logistics checks, admin pages, and ticketing updates rather than just summarizing docs.

Replace customers' ticketing platforms

Rejected because: Zendesk, Intercom, and similar tools already provide human operations UI, queues, and triage, so Lorikeet chose to integrate as an AI teammate.

Give customers a workflow builder and large prompt boxes as the main configuration interface

Rejected because: Customers struggled to know what to put in prompts and building-block workflows were not always easy for support teams to configure.

Fully hand off every blocked ticket to a human

Rejected because: Resolution-in-the-loop lets the agent keep ticket ownership when it only needs a narrow missing fact or policy decision.

Patterns

multi-agent-orchestration (agent-architecture)

Lorikeet separates a customer-facing concierge from an internal coach agent, allowing each to optimize for different risk levels and collaboration modes.

human-in-the-loop-gate (control)

The concierge defaults to human handoff when uncertain and uses resolution-in-the-loop to ask humans for targeted unblockers while retaining the ticket.

guardrails-io-filtering (safety)

Lorikeet runs isolated LLM guardrail checks before customer responses, using business context to steer or escalate when a response violates rules.

eval-driven-development (evaluation)

The product asks customers to define good and bad test cases first, then uses coach to build or adjust SOPs, simulations, and guardrails from those examples.

observability-tracing (operations)

Lorikeet exposes trace and error-analysis information to customers, then uses coach to diagnose why the AI behaved incorrectly and propose fixes.

confidence-thresholding (routing)

The system explicitly asks models to evaluate confidence and routes uncertain or risky cases to escalation or guardrail steering.

Tradeoffs

Integrate into existing ticketing systems

Chose An AI teammate/persona inside Zendesk, Intercom, and similar platforms over Replacing the customer's support operations platform.

Rationale: Ticketing systems already handle human operations workflows, and Lorikeet wanted to focus on conversational AI quality and collaboration.

Begin with email

Chose Email as the first automation channel over Starting with lower-latency live chat or voice.

Rationale: Email's forgiving response-time expectations gave the team room to connect tooling and resolve real issues before moving to more real-time channels.

Use resolution-in-the-loop

Chose Agent keeps ownership while asking humans for narrow unblockers over Either full automation or full escalation for blocked tickets.

Rationale: Many cases are just shy of AI resolution; a targeted human answer can unblock the agent without making the customer wait for a complete handoff.

Combine conversational coach with structured UI

Chose Hybrid UI with contextual entry points and review surfaces over A blank chatbot or a purely traditional configuration app.

Rationale: Chat helps express intent, but users need context-specific buttons and structured review to know where to start and verify the agent's work.

Gotchas (what bit them)

Positive feedback can hide that prototypes are not solving a burning problem.

Root cause: Early ops/support coaching and information-flow prototypes were liked but became another task for already busy teams.

Resolution: The team embedded with a healthcare support team and followed the demand to clear tickets directly.

Support automation fails if it is treated as FAQ retrieval.

Root cause: Real support tickets require browser/admin lookups, logistics checks, internal docs, and ticket-system actions.

Resolution: Lorikeet built toward agents that can use tools, act in ticketing systems, and ask for missing information.

Prompt boxes and workflow builders ask too much of non-technical support teams.

Root cause: Customers did not know what to put in large prompts, and building-block workflow UIs were hard to configure.

Resolution: Lorikeet moved toward coach-mediated conversational configuration plus structured review UI.

Generic guardrails can misfire in domain-specific regulated contexts.

Root cause: A hardcoded hostility guardrail treated a legitimate medical-cannabis question as hostile because it lacked business context.

Resolution: Guardrails became customizable and context-aware per customer business and regulatory environment.

Helpful customer-support personas are prone to overpromising.

Root cause: LLMs are fine-tuned to be helpful and become even more eager when prompted as customer support agents.

Resolution: Lorikeet adds confidence evaluation, LLM-as-judge checks, and guardrails before responses go out.

Evaluation

Lorikeet evaluates through real customer data, simulations/test cases, trace review, customer-defined good/bad examples, guardrail checks, and coach-assisted diagnosis of failures.

Metrics: Real customer email CSVs processed through the early CLI/spreadsheet prototype. · Repeated reading of 50 to 100 conversations/traces to identify behavior patterns. · Simulation scenarios for refund flows, escalation/non-escalation, and guardrail behavior. · Escalation distribution such as 50% of handoffs mapping to a knowledge gap. · Average handle time for human agents increasing as routine tickets are automated.

Outcomes

• Lorikeet moved from CLI/spreadsheet prototypes to self-service, traceable support automation.
• The product now includes a customer-facing concierge agent and an internal coach agent.
• Mature deployments shift human work toward harder tickets, increasing average handle time for human operators.
• Customers can improve SOPs, knowledge bases, guardrails, and simulations through coach and trace feedback.

The source describes multi-channel support, regulated customers, several early subscribers, mature companies at altitude, and escalation analytics, but it does not provide a single ticket-volume total.