Production Intent Recognition System for Enterprise Chatbots

feedyou · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise chatbot intent recognition

FeedYou and Prometeus AI needed production chatbot platforms that could route short, noisy user utterances through intent models and dialog flows while supporting rapid bot-specific iteration and modular conversation design.

Constraints:

• Most production inputs were only one or two words, limiting context for sophisticated models.
• Each chatbot had little training data and its own model rather than a huge shared dataset.
• False-positive intent matches frustrate users more than honest fallback.
• Dialog systems need versionable modular flows for production chatbot maintenance.

Architecture

FeedYou uses lightweight NLP.js intent classifiers per chatbot, with normalization/tokenization/stemming, confidence scoring, named entity extraction, and real-time validation during bot design. Similar intents are consolidated and disambiguated with entities and clarifying questions. Prometeus Flowstorm adds hierarchical sub-dialogs with local/global/out-of-domain intent recognition and GPT-2 generation for out-of-domain handling, evolving from Alexa Prize lessons about modular dialog management.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  feedbot-designer["Feedbot Designer"] --> nlp-js-trainer["NLP.js trainer"]
  feedbot-designer["Feedbot Designer"] --> dialog-tree-runtime["Dialog tree runtime"]
  nlp-js-trainer["NLP.js trainer"] --> validation-system["Validation system"]
  validation-system["Validation system"] --> chatbot-designer["Chatbot designer"]
  chatbot-designer["Chatbot designer"] --> feedbot-designer["Feedbot Designer"]
  dialog-tree-runtime["Dialog tree runtime"] --> intent-classifier["Intent classifier"]
  dialog-tree-runtime["Dialog tree runtime"] --> entity-recognizer["Entity recognizer"]
  intent-classifier["Intent classifier"] --> entity-recognizer["Entity recognizer"]
  entity-recognizer["Entity recognizer"] --> clarification-flow["Clarification flow"]
  clarification-flow["Clarification flow"] --> dialog-tree-runtime["Dialog tree runtime"]
  flowstorm-sub-dialog-manager["Flowstorm sub-dialog manager"] --> gpt-2-out-of-domain-handler["GPT-2 out-of-domain handler"]

Components

Data flow

A chatbot designer defines intents/examples in Feedbot Designer. When edited, the platform trains an NLP.js model in seconds, tests training examples, warns about overlapping intents, and deploys the bot to web/Facebook channels. At runtime, short user input is normalized, tokenized, classified, and entity-tagged. If intent/entity confidence is sufficient, the dialog tree routes to a branch; otherwise the bot asks clarification or falls back. Flowstorm similarly routes through local, global, and out-of-domain intent layers inside reusable sub-dialogs.

Why this architecture

FeedYou chose lightweight per-bot classifiers because production inputs were short, datasets were small, training needed to be fast, and the platform was JavaScript-native. Named-entity disambiguation and validation reduce false positives, while Flowstorm’s hierarchy keeps larger conversational systems modular and manageable.

Models used: NLP.js neural classifier · FastText (evaluated) · GPT-2 generative models

Scale

• FeedYou reports about 72% of customer queries require no human intervention.
  “approximately 72% of customer queries handled by their chatbots don't require human intervention”summary, stated
• About half of users interact outside standard working hours.
  “about half of users interact with chatbots outside of standard working hours”summary, stated
• One month of QnA data had over 4,000 queries.
  “one month of data from a QnA bot with over 4,000 queries”summary, stated
• About 25% of inputs were noise.
  “Approximately 25% of inputs are essentially noise”summary, stated
• Model training takes about 3 seconds on Azure.
  “Model training is fast (approximately 3 seconds on Azure), enabling rapid iteration”summary, stated
• The Alexa Prize work spanned four years and won the final year.
  “evolved from four years of competing in the Amazon Alexa Prize (winning the final year with their "Alquist" bot)”summary, stated

Alternatives considered (roads not taken)

FastText for intent matching

Rejected because: It added little value with one- or two-word inputs and tiny per-bot datasets.

Splitting intents across multiple models

Rejected because: It improved some targeted matches but created high-confidence false positives.

Separate intents for forgotten/new password

Rejected because: The model struggled to distinguish close intents; a shared password intent plus entity worked better.

Visual-flow export to code with redeploy cycles

Rejected because: Prometeus found the process slow and versioning-prone when the visual state was separate from code.

Patterns

confidence-thresholding (intent routing)

Intent confidence and similar-confidence warnings route ambiguous inputs to clarification/fallback rather than wrong answers.

structured-output-contract (dialog design)

Dialog trees and entity slots constrain how recognized intents map to branches.

eval-driven-development (authoring)

Every model edit triggers training and validation against examples for immediate author feedback.

fallback-degradation (conversation handling)

The system prefers clarification or “I do not understand” over high-confidence wrong intent matches.

prompt-chaining (dialog flow)

Flowstorm uses hierarchical sub-dialogs and multiple intent layers rather than one monolithic conversation model.

Tradeoffs

Model sophistication

Chose NLP.js lightweight classifiers over FastText/complex models.

Rationale: Small data, short inputs, JavaScript stack, and fast retraining mattered more than richer embeddings.

Intent design

Chose Consolidated intents plus entities over Many narrowly separated intents.

Rationale: Entities reduce classifier burden and produce more natural clarification flows.

Error posture

Chose Fallback/clarification over Risking high-confidence wrong answers.

Rationale: Users prefer being asked to rephrase over receiving an incorrect response.

Gotchas (what bit them)

Longer inputs matched worse than short ones.

Root cause: Production users mostly typed short/noisy phrases, and long inputs did not fit the intent-model assumptions.

Resolution: Choose models and validation around real input length distribution.

High-confidence wrong matches were worse than fallback.

Root cause: Splitting models narrowed vocabulary and inflated confidence for accidental token overlaps.

Resolution: Avoid multi-model splits that create false positives; consolidate intents and clarify.

Separate similar intents confused the classifier.

Root cause: Short inputs did not reliably distinguish semantic variants such as forgotten vs. new password.

Resolution: Use named entities inside a broader intent and ask clarifying questions when needed.

Evaluation

FeedYou evaluates by retraining in seconds on every edit, testing all training examples, warning on wrong/similar matches, and analyzing production input length/match-rate distributions.

Metrics: 72% no-human-intervention rate · outside-hours usage share · input length distribution · match success rate · training time · overlapping-intent warnings

Outcomes

• FeedYou chatbots handled about 72% of customer queries without human intervention.
• The validation loop gives designers immediate feedback on overlapping intents.
• Prometeus evolved Flowstorm from Alexa Prize lessons into hierarchical production dialog tooling.

The case reports 72% automated handling, about half of use outside working hours, 4,000+ monthly QnA queries in one analysis, 25% noise, 3-second Azure training, and four years of Alexa Prize work.