LLM-Powered Customer Support Agent Handling 50% of Inbound Requests

otter · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Restaurant-operations customer support automation

Otter needed an LLM-native support agent that could resolve high-volume restaurant support issues by diagnosing user intent, using internal systems safely, offering human escalation, and avoiding rigid decision-tree chatbot limits.

Constraints:

• Customer issues require deep integrations with menus, accounts, printer configuration, and other internal systems.
• Write operations need permission controls and explicit user confirmation to mitigate hallucination risk.
• Support must preserve the option to escalate to a 24/7 human agent.
• Vendor chatbots in Q1 2024 were either hard-coded decision trees or lacked the integration flexibility Otter needed.

Architecture

Otter built an in-house agentic support bot around function calling and plain-text runbooks. A GetRunbook function semantically matches the user's issue against a vector database of runbooks, an LLM chooses the right runbook or returns Not Found, and the bot follows runbook steps by asking follow-ups, invoking API/widget functions with the user's token, researching help articles for unmatched questions, requiring confirmation for critical writes, or escalating to a human agent through Zendesk.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Chat_interface["Customer chat interface"] --> Assistant_service["Otter Assistant service"]
  Assistant_service --> GetRunbook["GetRunbook function"]
  GetRunbook --> Runbook_vector_db["Runbook repository vector database"]
  GetRunbook --> Selection_LLM["Runbook selection LLM"]
  Assistant_service --> API_widgets["API and widget functions"]
  API_widgets --> Existing_APIs["Existing Otter APIs"]
  Existing_APIs --> Permission_model["Permission model"]
  API_widgets --> Confirmation_widget["Confirmation widget"]
  Assistant_service --> Research_function["Research function"]
  Research_function --> Knowledge_base["Support knowledge base"]
  Assistant_service --> Escalation["Human escalation path"]
  Evaluation["Evaluation framework"] --> Assistant_service
  Escalation --> Chat_interface

Components

Data flow

A restaurant user describes an issue. GetRunbook embeds and retrieves relevant runbooks, a separate LLM chooses the correct runbook or Not Found, and the bot executes the runbook by gathering information, calling APIs/widgets, researching help articles, or escalating to a human. Critical writes require explicit confirmation before the backend API executes under the user's permissions.

Why this architecture

Otter built in-house because support resolution needed deep, permissioned integrations and flexible LLM-native reasoning, while vendor tools were tied to decision trees or lacked the ability to manage complex resolution steps. Function calling let the agent mimic human support tasks while confirmation and escalation guarded risky actions.

Models used: LLM for intent matching · LLM for runbook selection · LLM support agent · embedding model · RAG on support knowledge base

Scale

• Otter Assistant handles roughly half of inbound customer requests without human intervention.
  “our in-house Gen-AI chatbot that currently handles ~50% of inbound customer requests quickly without human intervention.”article, stated
• The implementation was developed and scaled over about one year.
  “This post describes our journey over the past year building and scaling Otter Assistant.”article, stated
• Otter started vendor analysis in Q1 2024 and implementation in Q2 2024.
  “In Q1 2024, we started by analyzing the distribution of customer issues and found that resolving these tickets required deep integration with our systems.”article, stated

  “When we began implementation in Q2 of ‘24, the term “agentic” had yet caught on.”article, stated

Alternatives considered (roads not taken)

Established vendor chatbot products

Rejected because: They relied primarily on hard-coded decision trees and did not provide Otter's needed integration flexibility.

LLM-native startup support bots

Rejected because: Otter did not believe available startups could manage the complexity of the top resolution steps.

Hard-coded decision trees for bot logic

Rejected because: Otter wanted LLM-native, plain-text, modular runbooks that were easier to implement and traverse at runtime.

Allowing the LLM to execute critical writes directly

Rejected because: Hallucination risk required explicit user review and click confirmation before operations like pausing stores or updating prices.

Patterns

tool-use-orchestration (agent architecture)

Otter Assistant uses function calling across GetRunbook, API/widget functions, research, and human escalation to mimic human support workflows.

rag-retrieval (retrieval)

The runbook and research functions retrieve relevant support instructions and help articles from vector/search-backed stores before the LLM acts.

human-in-the-loop-gate (safety)

Critical writes require explicit user confirmation, and users can escalate to live agents when needed.

safety-policy-enforcement (governance)

Internal API calls reuse existing access-control models by passing the user's token through the service request.

eval-driven-development (evaluation)

Otter built a chatbot test/evaluation framework modeled on unit tests with inputs and expected outputs.

Tradeoffs

Build the support agent in-house

Chose custom Otter Assistant backend over third-party chatbot vendors.

Rationale: Otter needed flexible, permissioned integrations and complex resolution-step handling that vendors did not offer.

Represent automation logic as plain-text runbooks

Chose LLM-selected modular runbooks over hard-coded decision tree bot logic.

Rationale: Plain-text runbooks were easier to implement, maintain, and traverse dynamically during diagnosis.

Use confirmation widgets for writes

Chose user review and explicit click confirmation over fully autonomous write execution by the LLM.

Rationale: Confirmation mitigates hallucination and lets users modify inputs when the LLM gets details wrong.

Reuse existing APIs and authorization

Chose API wrapper calls with user tokens over new side-channel automation bypassing existing controls.

Rationale: Reusing internal APIs preserved permission boundaries while giving the agent real operational capability.

Gotchas (what bit them)

Vendor support bots could not provide both flexibility and integration depth.

Root cause: Existing products were either hard-coded decision trees or unlikely to handle the complex resolution steps required by top support issues.

Resolution: Build an in-house LLM-native backend while using Zendesk front-end pieces for MVP time-to-market.

Internal write operations created hallucination and authorization risk.

Root cause: The bot could modify real store/account state, so incorrect assumptions could have customer-visible consequences.

Resolution: Reuse existing permission models and require explicit user click confirmation for critical write operations.

Runbook matching can fail when no known issue pattern fits the user's request.

Root cause: Support questions may fall outside the top high-volume runbooks selected for automation.

Resolution: Return Not Found from GetRunbook and route to the research function or human escalation path.

Automation needed to preserve user trust in human support availability.

Root cause: Restaurant customers valued speed but still wanted access to 24/7 human agents.

Resolution: Keep seamless escalation to live agents as a first-class tool.

Evaluation

Otter built a chatbot evaluation framework that behaves like unit tests: define inputs, assert expected outputs, and use it for Otter Assistant and other chatbots.

Metrics: autonomous resolution share · customer satisfaction · unit-test-like expected-output assertions

Outcomes

• Otter Assistant resolves about half of support requests autonomously.
• The system keeps customer satisfaction intact according to the source.
• The architecture supports internal API actions, runbook diagnosis, research, and human escalation.

The agent was built and scaled over about one year, starting with Q1 2024 issue/vendor analysis and Q2 2024 implementation.