Evolving a Conversational AI Platform for Production LLM Applications

airbnb · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: customer support conversational AI platform

Airbnb needed to evolve Automation Platform from static step-by-step conversational workflows into an LLM-ready platform that could power customer-support applications while preserving workflow control, context management, tools, guardrails, and developer support.

Constraints:

Automation Platform v1 required teams to manually create workflows and tasks for every scenario.
LLM applications were still too new to fully rely on for millions of diverse customer experiences.
Sensitive claim-related products required strict validations and traditional workflows.
LLM applications needed context, tool execution, observability, guardrails, playgrounds, and insights.
The platform needed reusable guardrails and managed tool execution with retry or rate limiting.

Architecture

Automation Platform v2 combines traditional workflows with LLM-driven Chain of Thought. It prepares context, assembles prompts, sends them through an LLM adapter, executes requested tools through a Tool Manager, stores tool responses in runtime context, loops until a response is generated, and runs reusable guardrails in parallel across different tech stacks.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  inquiry-intake["Inquiry intake"] --> context-management["Context management"]
  context-management --> cot-io-handler["CoT IO handler"]
  cot-io-handler --> llm-adapter["LLM adapter"]
  llm-adapter --> tool-manager["Tool manager"]
  tool-manager --> runtime-context-manager["Runtime context manager"]
  runtime-context-manager --> guardrails-framework["Guardrails framework"]
  guardrails-framework --> inquiry-intake

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Inquiry intake`	Receives customer support messages and starts the LLM application flow.	`Context management`
`Context management`	Loads static or dynamic context such as chat history, customer role, trip information, and help articles.	`CoT IO handler`
`CoT IO handler`	Assembles prompt, contextual data, user input, and general processing before the LLM call.	`LLM adapter`
`LLM adapter`	Integrates customized logic and different LLM types into the platform.	`Tool manager`
`Tool manager`	Executes LLM-requested tools with payload preparation, retry, rate limiting, and observability.	`Runtime context manager`
`Runtime context manager`	Maintains updated runtime context across LLM calls and tool outcomes.	`Guardrails framework`
`Guardrails framework`	Runs reusable safety and execution guardrails in parallel.	`Inquiry intake`

Data flow

A user inquiry arrives, the platform collects chat history, user information, role, trip data, and other context, assembles a prompt, sends it to an LLM, executes any requested tool call such as fetching a reservation, updates context with the tool result, sends the new context back to the LLM, returns the final response, and records the conversation.

“Firstly, user inquiry arrives at our platform. Based on the inquiry, our platform collects relevant contextual information, such as previous chat history, user id, user role, etc.”article, stated

“Our platform follows this order and does the actual service call then saves call responses into the current context.”article, stated

Why this architecture

Airbnb chose a hybrid architecture because LLMs improved natural conversation and reasoning, but high-scale support still needed traditional workflows for sensitive tasks, deterministic tool execution, context control, and guardrails against hallucinations and jailbreaks.

“the best strategy is to combine them with traditional workflows and leverage the benefits of both approaches.”article, stated

“it is too early to fully rely on them for large scale and diverse experience for millions of customers at Airbnb.”article, stated

Models used: LLMs · Chain of Thought workflow · LLM Adapter · Tool Manager · Context Loader · Runtime Context Manager · Guardrails Framework

Scale

The platform is designed for Airbnb's millions of customers.
“large scale and diverse experience for millions of customers at Airbnb.”article, stated
Automation Platform v1 supported a suite of conversational AI products.
“Automation Platform v1, an enterprise-level platform developed by Airbnb to support a suite of conversational AI products.”article, stated

Alternatives considered (roads not taken)

Static predefined workflows only.

Rejected because: Manual workflow creation for every scenario was hard to scale and error-prone.

“product creators need to manually create workflows and tasks for every scenario, and repeat the process for any new use case later”article, stated

LLM-only workflow for all support tasks.

Rejected because: Sensitive claim workflows still needed strict validation and traditional processing.

“it’s more suitable to use a transition workflow instead of LLM to process a claim related product that requires sensitive data and numbers of strict validations.”article, stated

Static context declarations only.

Rejected because: Some use cases need dynamic context retrievers such as relevant help articles.

“developers are allowed to either statically declare the needed context (e.g. customer name) or name a dynamic context retriever”article, stated

Patterns

`prompt-chaining` (Chain of Thought workflow)

Automation Platform v2 loops through context preparation, LLM reasoning, tool execution, and updated context until a result is generated.

“Chain of Thought will stay in the reasoning loop until a result is generated.”article, stated

`tool-use-orchestration` (support actions)

The LLM reasons about which tools to use and in which order, while the platform executes those tools.

“use an LLM as the reasoning engine to determine which tools to use and in which order.”article, stated

`guardrails-io-filtering` (runtime guardrails)

Reusable guardrails monitor LLM communication and prevent unsafe tool execution.

“a safe-guarding mechanism that monitors communications with the LLM, ensuring it is helpful, relevant and ethical.”article, stated

`observability-tracing` (LLM interaction insights)

The platform adds LLM-oriented observability for detailed interaction insights such as latency and token usage.

“LLM-oriented observability with detailed insights into each LLM interaction, like latency and token usage.”article, stated

Tradeoffs

Use hybrid LLM and traditional workflow architecture.

Chose LLM reasoning plus traditional workflow controls over Fully LLM-driven support automation.

Rationale: The hybrid strategy gets more natural conversation while keeping strict validations for sensitive workflows.

“We believe that at this moment, the best strategy is to combine them with traditional workflows”article, stated

Represent traditional actions as tools.

Chose Existing actions and workflows as Chain of Thought tools over Building a separate tool execution system from scratch.

Rationale: Actions and workflows already had unified interfaces and managed execution environments.

“Actions and workflows work well as tools in Chain of Thought because of their unified interface and managed execution environment.”article, stated

Let developers choose static or dynamic context.

Chose Static declarations plus dynamic retrievers over A single context-loading strategy.

Rationale: Different support use cases need different context sources and point-in-time retrieval behavior.

“For use cases like offline evaluation, point-in-time data retrieval is also supported by our system via configuration.”article, stated

Gotchas (what bit them)

LLMs are not ready to own all support workflows.

Root cause: Production-level latency, hallucination, sensitive data, and strict-validation issues remain.

Resolution: Combine LLM applications with traditional workflows and guardrails.

“the community is improving some of its aspects to meet production level requirements, like latency or hallucination.”article, stated

Static workflows do not scale to every scenario.

Root cause: Product creators have to manually author and repeat workflows and tasks.

Resolution: Use LLM reasoning with tool execution on top of reusable platform components.

“Hard to scale: product creators need to manually create workflows and tasks for every scenario”article, stated

Tool execution needs platform controls.

Root cause: LLM-requested actions can require retries, rate limiting, observability, and bad-execution prevention.

Resolution: Use Tool Manager and tool guardrails rather than letting the LLM execute actions directly.

“Tool Manager: prepare tool payload with LLM input & output, manage tool execution and offer quality of life features like retry or rate limiting.”article, stated

Evaluation

The platform supports developer insights, LLM observability, latency and token usage tracking, guardrail execution, tool observability, and future LLM application simulation.

Metrics: Latency · Token usage · Guardrail execution result · Tool execution observability · Retry/rate-limit events · Conversation records · Developer insights

“context management, guardrails, playground and insights.”article, stated

“investigate LLM application simulation.”article, stated

Outcomes

Automation Platform v2 supports LLM applications for customer-support agents.
Developers can build applications using context management, Chain of Thought workflows, tools, guardrails, playgrounds, and insights.
The platform helps support agents work more efficiently, provide better resolutions, and respond faster.
Airbnb plans to expand agent frameworks, tool capabilities, and LLM application simulation.

The article does not disclose specific quantitative production metrics beyond referencing millions of Airbnb customers; scale is primarily platform scope and product domain.

“help customer support agents work more efficiently, provide better resolutions, and quicker responses.”article, stated

“we will continue to evolve with these transformative technologies, explore other AI agent frameworks”article, stated