Multilingual Content Navigation and Localization System

intercom · Media & Entertainment · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support automation

Intercom needed to move from curation-heavy traditional chatbot automation toward LLM-powered support experiences that could answer from knowledge bases with less setup friction and fewer hallucinations.

Constraints:

Resolution Bot required manual curation of question clusters and answers.
Customer-facing bots carry higher hallucination risk than agent-assist features.
Different LLM features have different error costs and context needs.
The team wanted rapid time-to-first-customer-use after ChatGPT launched.

Architecture

Intercom first shipped low-risk agent-assist features using GPT-4, then built Fin as a RAG-based customer-facing bot that answers support questions from knowledge bases. The development style starts with powerful models, ships broad features quickly, then optimizes cost and context.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-inbox-features["Support Inbox Features"]
  fin["Fin"]
  knowledge-base-retrieval["Knowledge Base Retrieval"]
  resolution-bot-baseline["Resolution Bot Baseline"]
  human-support-escalation["Human Support Escalation"]
  gpt-4["GPT-4"]
  support-inbox-features --> gpt-4
  fin --> knowledge-base-retrieval
  fin --> human-support-escalation
  knowledge-base-retrieval --> fin
  resolution-bot-baseline --> human-support-escalation
  human-support-escalation --> fin
  human-support-escalation --> resolution-bot-baseline
  gpt-4 --> support-inbox-features
  gpt-4 --> fin

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Support Inbox Features`	Provide summarization, tone adjustment, and text expansion for support representatives.	`GPT-4`
`Fin`	Customer-facing LLM-powered support bot.	`Knowledge Base Retrieval`, `Human Support Escalation`
`Knowledge Base Retrieval`	Provides grounded support context for Fin answers.	`Fin`
`Resolution Bot Baseline`	Traditional ML chatbot using curated question clusters and answers.	`Human Support Escalation`
`Human Support Escalation`	Handles unanswered or failed bot interactions.	`Fin`, `Resolution Bot Baseline`
`GPT-4`	Initial powerful model used for rapid production feature development.	`Support Inbox Features`, `Fin`

Data flow

For agent-assist, support inbox content is summarized, rewritten for tone, or expanded by GPT. For Fin, customer questions retrieve relevant knowledge-base material and the LLM generates grounded support answers with escalation when needed.

“By leveraging GPT-4 with retrieval-augmented generation (RAG), they achieved comparable accuracy rates without requiring extensive manual curation.”summary, stated

“later launched Fin, a customer-facing bot that draws from knowledge bases to answer support queries with minimal hallucination.”summary, stated

Why this architecture

Intercom used GPT-4 first because the fastest path to useful customer value was proving product behavior with a powerful model, especially for low-error-cost features, before cost optimization.

“Their development approach emphasizes breadth-first iteration, rapid time-to-first-customer-use, and starting with powerful but expensive models like GPT-4 before optimizing for cost.”summary, stated

“the ability to ship production features in weeks is genuinely new.”summary, stated

Models used: GPT-4 · GPT-3.5 · T5 · BERT

Scale

Intercom shipped initial ChatGPT-era prototypes by December 20 after deciding to build on December 5.
“they decided to build on December 5th, had prototypes by December 20th”summary, stated
Customers got access January 13 and public launch happened January 31.
“gave customers access on January 13th, and launched publicly with customer testimonials on January 31st”summary, stated
Summarization shipped in about a week.
“Summarization exemplifies this—GPT performs well, and occasional imperfections don't cause significant harm.”summary, implied

Alternatives considered (roads not taken)

Traditional ML Resolution Bot

Rejected because: It required heavy manual curation and was less accurate overall.

“The curation process was friction-heavy, and while accuracy was acceptable for simple questions, it wasn't great overall.”summary, stated

Immediate customer-facing LLM without retrieval grounding

Rejected because: The team initially feared hallucinations in direct customer interactions.

“Initially, the team avoided customer-facing LLM applications due to hallucination concerns—they believed "large language models will hallucinate too much" to put directly in front of customers.”summary, stated

Patterns

`rag-retrieval` (retrieval)

Fin grounds support answers in customer knowledge bases.

“draws from knowledge bases to answer support queries with minimal hallucination.”summary, stated

`human-in-the-loop-gate` (governance)

Traditional and LLM bots escalate unresolved support interactions to human agents.

“Users could accept the answer, request alternatives, or escalate to a human agent—the latter being the "failure state" they tried to minimize.”summary, stated

`model-routing` (architecture)

Intercom starts with powerful models for launch and later optimizes cost and context by feature difficulty.

“starting with powerful but expensive models like GPT-4 before optimizing for cost.”summary, stated

Tradeoffs

Start with agent-assist features

Chose Summarization, tone adjustment, and expansion in the inbox over Customer-facing bot first.

Rationale: Lower error costs made them safer to ship quickly.

“all focused on the inbox (agent-facing) rather than customer-facing applications:”summary, stated

Use GPT-4 early

Chose Powerful expensive model for fast product learning over Premature cost optimization.

Rationale: The team prioritized time-to-first-customer-use and product confidence.

“get features live with customers quickly”summary, stated

Gotchas (what bit them)

Curation-heavy bots impose ongoing setup friction.

Root cause: Traditional ML bots require pre-curated answers and cluster-boundary management.

Resolution: Use RAG-backed LLM answers to reduce manual curation while preserving grounding.

“The system worked by recognizing user questions and matching them to pre-curated responses.”summary, stated

LLM feature difficulty depends on both accuracy and error cost.

Root cause: Customer-facing support errors carry more harm than internal agent-assist imperfections.

Resolution: Ship low-risk inbox features first and treat customer-facing bots as harder.

“The speaker offers a useful framework for categorizing LLM feature difficulty based on two dimensions: out-of-the-box accuracy and cost of errors.”summary, stated

Evaluation

Intercom judged features by accuracy, error cost, customer usage, hallucination risk, and time to ship.

Metrics: accuracy · time-to-customer-use · error cost · hallucination rate

“Easy features combine high out-of-the-box accuracy with low error costs.”summary, stated

Outcomes

Intercom shipped ChatGPT-era inbox features within weeks.
Fin replaced heavy curation with RAG-backed support answers.
The team achieved comparable accuracy without extensive manual curation.

The launch timeline moved from December 5 decision to January 31 public launch.

“achieved comparable accuracy rates without requiring extensive manual curation.”summary, stated