Smart Ticket Routing and Support Agent Copilot using LLMs

adyen · Finance · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise payment support operations

Adyen needed to scale support operations as merchant and transaction volume grew, especially by reducing ticket handoffs and helping agents answer customer inquiries faster.

Constraints:

• Support pressure increased with merchant growth and transaction volume.
• Passing tickets between teams was a major factor in response time.
• Support suggestions needed current and relevant public/private documentation.
• The team wanted model flexibility, performance evaluation, and Kubernetes-based event-driven production integration.

Architecture

Adyen built two LLM applications with LangChain and LangSmith: a smart ticket router that analyzes ticket theme and sentiment, dynamically updates priority, and routes tickets to suited specialists; and a support-agent copilot that uses a vector database over public and private documents to retrieve up-to-date context and generate easily modifiable Question Answering Suggestions through an LLM integrated with Adyen's internal LLM API and Kubernetes microservice architecture.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Customer support tickets"] --> B["Smart ticket router"]
  B --> C["Specialist support teams"]
  A --> G["Support agent copilot"]
  D["Public and private documents"] --> E["Embedding model"]
  E --> F["Vector database"]
  F --> G
  G --> H["Internal LLM API endpoint"]
  H --> I["Kubernetes microservices"]
  G --> C
  J["LangSmith evaluation"] --> G

Components

Data flow

Incoming tickets are analyzed for theme, sentiment, user-based priority, and routing destination. For answer suggestions, public and private documents are embedded into a vector database, retrieved for each ticket, passed to an LLM through Adyen's internal endpoint, and returned as modifiable response suggestions for support agents.

Why this architecture

Adyen split routing from answer generation because handoffs and response drafting were separate support bottlenecks, and used LangChain/LangSmith to preserve model flexibility while evaluating response quality and costs.

Models used: LLMs · embedding model · LangChain · LangSmith

Scale

• Adyen supports large companies including Meta, Uber, H&M, and Microsoft.
  “large companies like Meta, Uber, H&M, and Microsoft”article, stated
• The team built the document collection and first copilot milestone in four months.
  “In just 4 months the Adyen team was able to build a comprehensive collection of relevant documents”article, stated
• The initial team was a lean group of data scientists and ML engineers in Madrid.
  “a new and lean team of Data Scientists and Machine Learning Engineers based out of their new Tech Hub in [Madrid, Spain]”article, stated

Alternatives considered (roads not taken)

Increase support headcount

Rejected because: Adyen wanted to scale support through technology without increasing team size.

Traditional keyword search over documents

Rejected because: The vector retrieval milestone far outperformed keyword search and built trust.

Single fixed model integration

Rejected because: Adyen wanted to experiment and switch among LLMs easily.

Patterns

rag-retrieval (retrieval)

The support copilot retrieves relevant public and private documents from a vector database before generating suggested answers.

model-routing (model-selection)

LangChain lets Adyen switch the core LLM in the chain during experimentation.

observability-tracing (observability)

LangSmith evaluates application performance and compares model quality and cost.

tool-use-orchestration (agent-architecture)

Ticket routing and copilot workflows orchestrate LLMs, retrieval, internal endpoints, and Kubernetes microservices.

Tradeoffs

Build with LangChain and LangSmith.

Chose customizable framework plus evaluation platform over single-model bespoke integration.

Rationale: The team wanted prototype-to-production speed, model flexibility, and cost/quality comparisons.

Keep suggested answers modifiable.

Chose agent copilot suggestions over fully automated support replies.

Rationale: Support agents remain in the loop and can edit proposed customer responses.

Gotchas (what bit them)

Ticket handoffs slowed support response.

Root cause: Tickets passed between teams before reaching the right expert.

Resolution: Analyze content, theme, sentiment, and priority to route tickets to suited specialists.

Support answer generation depends on fresh, trusted documents.

Root cause: Support responses require relevant and up-to-date public/private documentation.

Resolution: Build a document collection with vector retrieval before connecting the LLM answer generator.

Model choice affects both quality and cost.

Root cause: Different LLMs produce different response quality/cost tradeoffs.

Resolution: Use LangSmith to evaluate model changes and LangChain to switch models.

Evaluation

Adyen uses LangSmith to evaluate LLM application performance and compare model effects on response quality and costs; the source reports qualitative support-efficiency outcomes rather than detailed quantitative metrics.

Metrics: response quality · model cost · routing accuracy · response time · support agent satisfaction

Outcomes

• Smart ticket routing became more efficient and accurate.
• Document retrieval outperformed traditional keyword search.
• Support agents received easily modifiable suggested answers.
• The initial system was built in four months.

The public source describes growth pressure, major enterprise merchants, a four-month build, and qualitative efficiency/satisfaction gains but does not provide ticket-volume metrics.