Scaling Chatbot Platform with Hybrid LLM and Custom Model Approach

voiceflow · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Hybrid conversational AI platform with LLM and custom model workloads

Voiceflow needed to add generative AI features to a chatbot/voice assistant platform while preserving low-latency, low-cost custom NLU for intent and entity tasks.

Constraints:

• Generative outputs needed JSON parsing and error handling.
• LLM providers had inconsistent latency and availability characteristics.
• Self-hosting LLMs was not core to Voiceflow's business, but custom NLU was core.
• Runtime customer conversations need lower latency and reliability than creation-time playground features.
• Fine-tuning, few-shot prompting, APIs, and custom models each carry different cost and operational tradeoffs.

Architecture

Voiceflow built an ML Gateway that exposes a unified interface over custom language models and third-party LLM providers such as OpenAI, Azure OpenAI, Anthropic Claude, and Claude Instant. The gateway handles prompt validation, rate limiting, usage tracking, and provider abstraction. Voiceflow kept custom self-hosted NLU for core intent/entity workloads, used provider APIs for generative features, added regex/prompt handling for JSON, benchmarked provider latency, and refactored custom-model serving when Pub/Sub caused p99 spikes.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Voiceflow product features"] --> B["ML Gateway"]
  B --> C["Provider LLM APIs"]
  B --> D["Custom NLU model service"]
  B --> E["Prompt validation layer"]
  B --> G["Usage tracking"]
  C --> F["JSON parsing cleanup"]
  F --> A
  D --> H["Redis queue"]

Components

Data flow

Creation-time and runtime product features call the ML Gateway. The gateway routes custom NLU requests to Voiceflow-hosted models and generative requests to provider APIs, applying prompt validation, rate limiting, and usage tracking. Outputs are parsed and cleaned when needed, while latency and cost benchmarks guide provider/model choices.

Why this architecture

Voiceflow uses LLM APIs for flexible generative capabilities while keeping specialized NLU in-house because the custom model is faster, cheaper, and more accurate for their core intent/entity workload.

Models used: OpenAI ChatGPT · Azure ChatGPT · Anthropic Claude · Claude Instant · Voiceflow custom NLU model · GPT-4

Scale

• Voiceflow's custom NLU model costs about 1000x less than GPT-4 for the tested inference workload.
  “it costs a thousand times more than than our model to actually do inference on”transcript, stated
• Azure ChatGPT was almost three times faster than regular ChatGPT for their benchmarked tasks.
  “found that Azure was almost three times faster um for for doing these kinds of tasks”transcript, stated
• The old Pub/Sub architecture targeted 150ms p50 in each direction but produced high p99 spikes for custom NLU.
  “we had an SLA of 150 milliseconds in each direction for p50s”transcript, stated
• The underlying language model itself returned in roughly 16-18ms.
  “the actual language model had very consistent latencies right it was between 16 and 18 milliseconds”transcript, stated

Alternatives considered (roads not taken)

Self-host LLMs for generative features

Rejected because: Self-hosting LLM research/infrastructure was not core to Voiceflow's business.

Use GPT-4 for custom NLU tasks

Rejected because: The custom NLU model outperformed GPT-4 on accuracy and cost for Voiceflow's specific task.

Keep Pub/Sub serving for latency-sensitive custom NLU

Rejected because: Pub/Sub caused high p99s even though the model itself had stable low latency.

Patterns

model-routing (model-ops)

Voiceflow routes tasks through an ML Gateway to custom NLU or third-party LLM providers depending on workload fit.

cost-aware-routing (optimization)

Custom NLU remains in-house because GPT-4 is far more expensive for the same core task.

structured-output-contract (interface)

Generated JSON required prompt engineering plus regex/handwritten cleanup rules.

latency-optimization (optimization)

Provider benchmarks and serving refactors addressed inconsistent latency and p99 spikes.

Tradeoffs

Generative feature integration

Chose third-party LLM APIs through ML Gateway over self-hosted LLM infrastructure.

Rationale: Voiceflow wanted to focus on platform features rather than LLM research and hosting.

Core NLU workload

Chose self-hosted custom NLU model over GPT-4 for intent/entity detection.

Rationale: The custom model was more accurate, lower latency, and dramatically cheaper for the task.

Gotchas (what bit them)

LLM JSON output was messy.

Root cause: Generative models did not reliably produce clean JSON formats.

Resolution: Add prompt engineering, regex, handwritten rules, and tests around format handling.

Provider latency varied enough to affect product decisions.

Root cause: OpenAI-style APIs had inconsistent latency and p99s across providers.

Resolution: Benchmark Azure ChatGPT versus regular ChatGPT and choose lower-variance providers where appropriate.

A serving architecture designed for one model class failed for another.

Root cause: The Pub/Sub design worked for earlier workloads but produced high p99s for latency-sensitive NLU.

Resolution: Refactor to Redis closer to the application layer.

Evaluation

Voiceflow evaluates through provider latency benchmarks, p50/p99 serving metrics, cost comparisons, accuracy tests against custom NLU, and production fit by use case.

Metrics: latency · p50 latency · p99 latency · standard deviation · inference cost · NLU accuracy

Outcomes

• Voiceflow integrated OpenAI, Azure ChatGPT, Claude, and Claude Instant through ML Gateway.
• Azure ChatGPT benchmarked nearly 3x faster for tested tasks.
• Voiceflow retained custom NLU because GPT-4 cost roughly 1000x more for the same inference task.
• A Pub/Sub serving bottleneck was refactored to Redis to meet latency targets.

The case emphasizes production routing and serving metrics rather than end-user volume; the clearest scale signal is the cost gap between GPT-4 and a custom model at production inference volume.