Scaling Customer Support AI Chatbot to Production with Multiple LLM Providers

intercom · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: customer support chatbot reliability

Intercom needed to run Finn, a revenue-bearing customer support bot, at production reliability while using multiple LLM providers and Bedrock infrastructure.

Constraints:

• Finn only earns revenue when it resolves a conversation, so failed or low-quality LLM calls directly hurt unit economics.
• The main bot flow contains many LLM calls, so a small per-call failure rate compounds into user-visible errors.
• Provider capacity, streaming behavior, prompt portability, and model-family differences all affect reliability and cost.

Architecture

Intercom runs Finn as a multi-step support bot with RAG-like LLM components, custom observability, prompt/input logging, offline LLM-as-judge evaluation, production A/B tests, Bedrock cross-region inference, selective provisioned throughput, and fallback model/provider strategies.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Finn support bot"] --> B["Amazon Bedrock runtime"]
  A --> C["Prompt and transaction logging"]
  A --> I["Observability and tracing"]
  C --> D["Offline evaluation engine"]
  D --> E["Production A/B test"]
  E --> A
  B --> F["Cross-region inference"]
  B --> G["Provisioned throughput"]
  B --> H["Fallback model router"]
  H --> F
  I --> C

Components

Data flow

Customer conversations flow through Finn's multi-call bot path. Prompt templates and prompt inputs are logged separately, replayed through candidate prompt or model changes, evaluated offline with LLM proxy metrics, then promoted to A/B tests. At runtime, Bedrock requests use retries, cross-region inference, provisioned throughput for selected workloads, and fallback models/providers when needed.

Why this architecture

The architecture treats LLM calls as unreliable production dependencies: log the right inputs for replay, evaluate changes offline, gate them through A/B tests, and make serving resilient with retries, regions, provisioned capacity, and fallbacks.

Models used: Amazon Bedrock models · Anthropic models · fallback model families · LLM-as-judge evaluators

Scale

• Finn had resolved more than 13 million conversations.
  “it has resolved more than 13 million conversations for more than 4,000 customers”transcript, stated
• Intercom reports more than 4,000 customers using the product.
  “it has resolved more than 13 million conversations for more than 4,000 customers”transcript, stated
• Many customers get more than 50% conversation resolution from the bot.
  “most of customers get more than 50% of resolution rate out of all the conversations is sent to our bot”transcript, stated
• Cross-region inference carried an observed overhead around 52 milliseconds.
  “there is I don't know 52 millisecond overhead on using cross region inference uh but you really have to test it for your use case and your model”transcript, stated
• Provisioned throughput saved about 20 milliseconds per token in one observed comparison.
  “for each Tok token we save around 20 milliseconds so for small prompts doesn't matter that much for prompts that generate very long responses we actually save several second”transcript, stated

Alternatives considered (roads not taken)

Streaming all model calls

Rejected because: Streaming enabled token-level metrics but made client retries difficult, so Intercom stopped using it everywhere for reliability-sensitive paths.

Pure on-demand Bedrock capacity

Rejected because: The shared pool could throttle before nominal quota, so selected workloads moved to provisioned throughput.

Single model family without fallbacks

Rejected because: Model availability failures are not perfectly correlated, but fallback models require prompt adaptation and testing.

Patterns

observability-tracing (operations)

Intercom tracks token-level latency metrics, retries, prompt inputs, and request traces to diagnose LLM serving health.

llm-as-judge (evaluation)

Candidate bot changes are replayed offline and scored by LLM prompts that estimate whether the answer resolves the user query.

fallback-degradation (reliability)

Requests can retry across regions, providers, and model families when one path fails.

model-routing (model-ops)

Intercom recommends choosing models by use case rather than assuming one model fits every workload.

Tradeoffs

Prompt logging design

Chose log templates and inputs separately over only logging materialized prompts.

Rationale: Separate logging lets the team replay production inputs against new prompt templates without conflating template, variables, and generation.

Capacity strategy

Chose selective provisioned throughput over all on-demand capacity.

Rationale: Provisioned throughput improved stability and latency for some workloads, but the large model units made blanket use expensive.

Serving resilience

Chose multiple regions and providers over single-region single-provider serving.

Rationale: Provider/model failures are imperfectly correlated, so fallback diversity improved availability despite added testing complexity.

Gotchas (what bit them)

Streaming made retry behavior worse.

Root cause: Streaming clients often do not retry failed model calls, so a partial or failed stream becomes user-visible.

Resolution: Be explicit about where streaming is actually needed and disable it on reliability-critical paths.

Published quota did not mean guaranteed capacity.

Root cause: On-demand Bedrock capacity used a shared pool that could throttle before the maximum token quota.

Resolution: Use provisioned throughput selectively where stable capacity and latency justify the cost.

Fallback models are not plug-compatible.

Root cause: Prompts often depend on model-family behavior and must be adapted before fallback paths can be trusted.

Resolution: Test fallback performance and use prompt optimization or manual adaptation before relying on secondary models.

Evaluation

Intercom combines offline replay with LLM proxy metrics and production A/B tests on real product metrics.

Metrics: time to first token · time to sample token · LLM-as-judge proxy resolution · production A/B product metrics · error rate · resolution rate

Outcomes

• Finn resolved more than 13 million conversations for more than 4,000 customers.
• Most customers get more than 50% resolution from conversations sent to the bot.
• After retries, cross-region inference, provisioned throughput, and provider/model fallback, Intercom reports most hours effectively have no errors.

The product is revenue-sensitive because Intercom charges only on resolved conversations, making reliability and unit cost core engineering concerns.