ML-Powered Interactive Voice Response System for Customer Support

airbnb · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time customer-support voice automation

Airbnb needed to replace rigid phone-menu support with a conversational IVR that understands caller issues in natural language, recommends relevant self-service resources, and escalates to humans when needed without adding awkward voice latency.

Constraints:

• Voice callers need low-latency responses; delays in ASR, intent detection, retrieval, or paraphrasing degrade the support experience.
• Generic speech recognition made domain-specific errors such as confusing listing with lifting.
• The system needed to preserve escalation paths for callers who ask for agents or whose issues require human intervention.
• Help article retrieval had to work across IVR, chatbot, and Help Center surfaces.
• Paraphrases needed predictable quality because callers cannot see article titles or contents before receiving links.

Architecture

Airbnb built an ML-powered IVR pipeline that prompts callers to describe issues in natural language, transcribes speech with Airbnb-specific ASR, classifies contact reason through the Issue Detection Service, and then either routes to self-service or escalates to a support agent with context. Self-service uses a two-stage Help Article Retrieval and Ranking system: article embeddings are indexed in a vector database, caller queries retrieve up to 30 candidates by cosine similarity, and an LLM-based ranker selects the top article for SMS or app delivery. A lightweight paraphrasing layer maps inquiries to curated standardized summaries by embedding similarity and a calibrated threshold so users hear what the system understood before receiving help content.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Caller["Caller voice input"] --> ASR["Airbnb-specific ASR"]
  ASR --> Intent["Issue Detection Service"]
  Intent --> Decision["IVR workflow decision"]
  Decision --> Retrieval["Help Article Retrieval and Ranking"]
  Retrieval --> Vector["Vector database"]
  Retrieval --> Para["Paraphrasing model"]
  Para --> Notify["SMS or app notification"]
  Decision --> Agent["Human support agent routing"]

Components

Data flow

A caller reaches Airbnb support and states their issue. Airbnb-specific ASR transcribes the audio, then an intent model predicts a Contact Reason. The IVR workflow decides whether to route the caller to self-service or a human agent. For self-service, the query is embedded, matched against Help Article embeddings in a vector database, re-ranked by an LLM model, summarized through a curated paraphrase, and sent by SMS or app notification.

Why this architecture

Airbnb chose a staged architecture because each support step has a different failure mode and latency budget: ASR must handle phone audio and domain terms, intent detection must be nearly imperceptible, article retrieval must search broadly but rank precisely, and paraphrasing must be predictable enough for voice callers. Human routing remains in the loop when automation is not appropriate.

Models used: Airbnb-specific ASR · Contact Reason Detection model · escalation intent detection model · text embedding model · LLM-based ranking model · paraphrasing nearest-neighbor model

Scale

• Airbnb reduced ASR word error rate from 33% to approximately 10%.
  “reduced the word error rate (WER) from 33% to approximately 10%”article, stated
• Intent detection averages under 50 ms latency.
  “latency remains under 50ms on average”article, stated
• Semantic retrieval returns up to 30 candidate articles.
  “retrieval of up to 30 relevant articles per user query using cosine similarity”article, stated
• Help article retrieval typically completes within 60 ms.
  “using cosine similarity, typically within 60ms.”article, stated
• Manual evaluation confirmed paraphrasing precision above 90%.
  “Manual evaluation of end-to-end model outputs confirmed precision exceeding 90%.”article, stated

Alternatives considered (roads not taken)

Use rigid menu-tree IVR

Rejected because: Rigid menus force callers through pre-set button paths instead of letting them describe issues naturally.

Use generic pretrained ASR unchanged

Rejected because: Generic ASR made domain-specific transcription errors that degraded downstream intent and retrieval.

Generate dynamic paraphrases with a generative LLM

Rejected because: Airbnb preferred curated summaries matched by embeddings to avoid latency and quality variability in voice prompts.

Patterns

tool-use-orchestration (agent_architecture)

The IVR workflow orchestrates ASR, intent detection, retrieval/ranking, paraphrasing, SMS/app delivery, and human escalation.

rag-retrieval (retrieval)

The system retrieves relevant Airbnb Help Center articles from embedded article content based on the caller's query.

retrieval-ranking (retrieval)

An LLM-based ranking model re-ranks semantically retrieved article candidates and presents the top-ranked article.

latency-optimization (performance)

Intent detection runs in parallel under 50 ms on average and retrieval typically completes within 60 ms to preserve real-time voice UX.

confidence-thresholding (quality_control)

The paraphrasing system uses a calibrated similarity threshold before selecting curated standardized summaries.

human-in-the-loop-gate (operations)

The IVR detects escalation requests or issues requiring intervention and routes callers to human support with context.

Tradeoffs

Use curated paraphrases instead of generative paraphrasing

Chose Nearest-neighbor matching to UX-written standardized summaries over Dynamic generative LLM paraphrasing.

Rationale: The curated approach favored predictable quality and low latency over open-ended generation in a voice flow.

Use two-stage article retrieval and ranking

Chose Vector retrieval for up to 30 candidates followed by LLM re-ranking over Directly presenting nearest-neighbor results without a ranker.

Rationale: The design balances retrieval breadth with ranking precision while meeting real-time latency requirements.

Respect explicit requests for human support

Chose Separate escalation intent detection and human routing over Forcing all callers through automation first.

Rationale: The system optimizes automation where appropriate while honoring callers who ask for an agent or have issues requiring intervention.

Gotchas (what bit them)

Domain-specific ASR errors cascade into intent and retrieval failures.

Root cause: General speech recognition models misrecognized Airbnb-specific support terms in noisy phone audio.

Resolution: Airbnb adapted ASR for noisy phone audio and added domain-specific phrase list optimization.

Voice support cannot hide slow model components behind a visual interface.

Root cause: Real-time voice delays create awkward pauses and degrade the user experience.

Resolution: Airbnb kept intent detection under 50 ms on average and semantic retrieval within about 60 ms.

Voice callers need confirmation before receiving invisible help content.

Root cause: Unlike visual interfaces, IVR callers cannot inspect article titles or contents before the system sends a link.

Resolution: A curated paraphrasing model summarizes the interpreted issue before presenting the article link.

Evaluation

Airbnb evaluates the IVR system at multiple layers: ASR word error rate, intent and retrieval latency, Precision@N for help article retrieval/ranking, manual precision for paraphrasing, user engagement with article content, self-resolution rates, NPS, and customer-service handling time.

Metrics: ASR word error rate from 33% to approximately 10% · intent detection latency under 50 ms on average · semantic retrieval within 60 ms · Precision@N for retrieval and ranking · paraphrase precision exceeding 90% · user engagement with articles · self-resolution rates · customer NPS and handling time

Outcomes

• ASR word error rate dropped from 33% to approximately 10%.
• Intent detection latency remained under 50 ms on average.
• Semantic article retrieval typically completed within 60 ms.
• Manual paraphrasing evaluation exceeded 90% precision.
• Paraphrased summaries increased user engagement with article content and improved self-resolution rates.
• The shared help article retrieval/ranking system also supports Airbnb's chatbot and Help Center search.

The article gives component-level scale and latency numbers: WER from 33% to 10%, under-50 ms intent detection, up to 30 retrieved articles, retrieval within 60 ms, and paraphrase precision above 90%.