Building an LLM-Powered Support Response System

stripe · Finance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support response assistance

Stripe wanted to help support agents solve customer cases more efficiently, but out-of-the-box GPT responses were inaccurate for Stripe's complex support domain and offline metrics did not automatically translate to agent adoption.

Constraints:

Stripe support covers a large, complex product space with tens of thousands of weekly cases.
Customers still needed to talk to support agents, so the system assisted agents rather than replacing them.
Out-of-box GPT lacked current, specific Stripe support knowledge.
Online accuracy and adoption were hard to measure directly at production scale.

Architecture

Stripe built a sequential GPT support-assistance framework. Fine-tuned classification stages filter candidate support questions before a fine-tuned agent-response model generates information for support agents. The team emphasized online monitoring, proxy metrics, feedback loops, UX adoption work, and label-data quality over simply using larger models.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-case[Customer Case] --> classification-stages[Classification Stages]
  classification-stages --> agent-response-model[Agent Response Model]
  agent-response-model --> support-agent[Support Agent]
  classification-stages --> monitoring-dashboard[Monitoring Dashboard]
  agent-response-model --> monitoring-dashboard[Monitoring Dashboard]
  label-data-program[Label Data Program] --> classification-stages[Classification Stages]
  label-data-program --> agent-response-model[Agent Response Model]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Customer Case`	Incoming support request from a Stripe customer.	`Classification Stages`
`Classification Stages`	Fine-tuned filters for suitable support questions.	`Agent Response Model`
`Agent Response Model`	Fine-tuned generator for support-agent response assistance.	`Support Agent`
`Support Agent`	Human support worker who talks directly to customers.	`Agent Response Model`
`Monitoring Dashboard`	Online metrics and proxy feedback for shipped models.	`Classification Stages`, `Agent Response Model`
`Label Data Program`	Maintains the domain-specific training and evaluation data.	`Classification Stages`, `Agent Response Model`

Data flow

A support case enters the system, classification models decide whether it is suitable for an LLM-assisted response, a fine-tuned response model generates agent-facing information, and online monitoring/proxy metrics track whether the support workflow is actually improving.

“sequential GPT framework the first two fine-tuned classification steps filter any candidate user support questions”transcript, stated

“our fine-un fine-tuned agent response model which generates the information accurate response and these responses are finally adjusted to meet striped tone before being prompted to the agents”transcript, stated

Why this architecture

Stripe decomposed support response generation into classification and generation because the support domain was too broad for prompt engineering alone. Fine-tuning reduced hallucination, while online monitoring and UX work addressed the production gap that offline metrics missed.

“stripe support space is just too large for this to be a viable solution so we found that to solve problems that require deep subject matter expertise at scale we need to break down the problem into more ml tangible steps”transcript, stated

“fine-tuning which in our case completely mitigated hallucinations also the other great benefit here is that fine-tuning on GPT requires just about 500 labels per class so we could move really quickly here relying exclusively”transcript, stated

Models used: DaVinci GPT · fine-tuned GPT classification models · fine-tuned agent response model

Scale

Stripe support agents handle tens of thousands of support cases every week.
“operations org is very very busy every week our Asians handle tens of thousands of support cases where they answer questions and solve problems for our customers so as the majority of this support volume is text-based data llms are an”transcript, stated
Fine-tuning required about 500 labels per class.
“fine-tuning on GPT requires just about 500 labels per class so we could move really quickly here relying exclusively on Expert agent annotations for our pilot our end solution consists of a”transcript, stated
Online case labeling was not feasible at production scale.
“unfortunately because online case labeling was not feasible at this scale we had a considerable Gap when it came to unserving online accuracy trends so once we shift we realize that the rate of which agents were using our ml”transcript, stated

Alternatives considered (roads not taken)

Use out-of-box GPT for support answers.

Rejected because: It gave inaccurate answers for basic support questions.

“when we ask out of box DaVinci GPT a basic support question like hey how can I pause payouts GPT will give a very plausible sounding answer but unfortunately it is actually incorrect and this is true for the majority of”transcript, stated

Rely on prompt engineering alone.

Rejected because: Stripe's support space was too large and complex.

“engineering the scope and complexity of stripe support space is just too large for this to be a viable solution so we found that to solve problems that require deep subject matter expertise at scale we need to break down the problem”transcript, stated

Trust offline testing alone.

Rejected because: Production showed adoption and online monitoring gaps despite positive offline feedback.

“so even though our offline user testing feedback was really positive and our online ml accuracy Trends were good in practice agents were just too accustomed to their flow of solving cases so ignoring our prompts and this lack of agent engagement was a huge bottleneck”transcript, stated

Patterns

`fine-tuning-adaptation` (support)

Stripe fine-tuned GPT classifiers and response models on support labels.

“fine-tuned classification steps”transcript, stated

`prompt-chaining` (workflow)

The system breaks support response assistance into classification and generation steps.

“sequential GPT framework”transcript, stated

`observability-tracing` (operations)

Stripe prioritized online monitoring and dashboards before shipping.

“not shipped unless it has full monitoring in a dashboard”transcript, stated

`data-quality-loop` (training data)

The team found label-data quality more valuable than model sophistication.

“label data quality yielded a lot higher performance gains compared to using more”transcript, stated

Tradeoffs

System decomposition

Chose Sequential classification plus generation over One-shot answer generation.

Rationale: Breaking the problem into manageable ML steps made the support task tractable.

“more ml manageable steps”transcript, stated

Monitoring priority

Chose Full online monitoring before shipping over Adding dashboards later.

Rationale: Online feedback was essential for knowing whether the system solved the real problem.

“monitoring is compromised as something like we can catch up on later”transcript, stated

Data strategy

Chose Better label data and weak supervision over Only larger or more sophisticated models.

Rationale: Label-data iteration yielded larger gains and could scale better than manual generative labels.

“leverage weak supervision techniques like snorkel machine learning”transcript, stated

Gotchas (what bit them)

Offline success can hide adoption failure.

Root cause: Support agents were accustomed to existing workflows and production usage differed from offline tests.

Resolution: Treat UX adoption and online proxy metrics as first-class work.

“in practice agents were just too accustomed”transcript, stated

Missing online accuracy labels can leave teams blind after launch.

Root cause: Online case labeling was infeasible at support scale.

Resolution: Develop heuristic match-rate proxy metrics and dashboards.

“we developed a heuristic based match rate to represent”transcript, stated

Generic model errors often reflect domain data gaps.

Root cause: Many errors came from Stripe-specific support gotchas, not general model weakness.

Resolution: Invest in domain label data quality and weak supervision for scalable labels.

“gotchas in the Striped support space as opposed to more General gaps”transcript, stated

Evaluation

Stripe used offline black-box evaluations during development, then added online proxy metrics, monitoring dashboards, agent adoption analysis, and label-quality iteration after production gaps appeared.

Metrics: Offline ML performance · Online heuristic match rate · Agent adoption · Online accuracy trends · Label quality

“directional feedback using heuristics is a million times better”transcript, stated

Outcomes

Fine-tuning mitigated hallucinations in the support-response workflow.
Initial production deployment exposed low adoption despite positive offline testing.
Stripe elevated online monitoring, proxy metrics, UX work, and label-data quality as core lessons.

The transcript reports tens of thousands of weekly support cases, about 500 labels per class for fine-tuning, and production-scale online labeling constraints.

“customers will always talk directly to our support agents”transcript, stated