AI-Powered Chatbot Automation with Hybrid NLU and LLM Approach

scotiabank · Finance · 2022 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Banking chatbot sustainment and handoff summarization

Scotiabank needed to improve and maintain a large intent-based chatbot while reducing manual trainer review effort and helping live agents understand long chatbot conversations quickly.

Constraints:

• financial institutions require tight control over chatbot responses
• the bot covers over 700 response types
• manual query review consumed recurring AI trainer effort
• training data changes can affect model accuracy
• LLM summarization must avoid hallucinating transaction details

Architecture

Scotiabank added an AI-for-AI sustainment layer to a controlled NLU chatbot: Luigi, Eva, and Peach review misclassifications and training data, while a separate LLM summarizer condenses chats for live-agent handoff.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  intent-based-chatbot["Intent-based chatbot"]
  conversation-logs["Conversation logs"]
  luigi-classifier["Luigi classifier"]
  eva-classifier["Eva classifier"]
  peach-similarity-checker["Peach similarity checker"]
  training-phrase-database["Training phrase database"]
  llm-summarizer["LLM summarizer"]
  live-agent-handoff["Live-agent handoff"]
  intent-based-chatbot --> conversation-logs
  conversation-logs --> luigi-classifier
  conversation-logs --> eva-classifier
  conversation-logs --> peach-similarity-checker
  luigi-classifier --> training-phrase-database
  eva-classifier --> training-phrase-database
  peach-similarity-checker --> training-phrase-database
  training-phrase-database --> intent-based-chatbot
  intent-based-chatbot --> live-agent-handoff
  llm-summarizer --> live-agent-handoff

Components

Data flow

Conversation data feeds classifiers for misclassification, intent suggestion, and training-phrase similarity, then curated phrases improve the chatbot; a separate LLM summary supports live-agent handoff.

Why this architecture

The design keeps the controlled NLU bot because banking responses need tight control, and LLM summarization needs safeguards against hallucinated transaction details.

Models used: binary classifier · multiclass classifier · similarity algorithm · text-bison · Gemini

Scale

• The chatbot covers more than 700 response types, and the AI sustainment models reportedly save almost a thousand hours annually.
  “our chatbot is able to provide over 700 responses and this is uh including different topics like banking accounts um investment credit cards loans mortgages and the most the main frequently asked questions”transcript, stated

  “altogether these models were able to save almost a, hours on an annual basis for us”transcript, stated
• The controlled chatbot covers over 700 response types.
  “our chatbot is able to provide over 700 responses and this is uh including different topics like banking accounts um investment credit cards loans mortgages and the most the main frequently asked questions”transcript, implied
• The sustainment models saved almost 1,000 hours annually.
  “altogether these models were able to save almost a, hours on an annual basis for us”transcript, stated
• The live-agent handoff summary reduces chatbot conversation length by about 80%.
  “this summary actually is able to reduce 80% the conversation so it saves um a lot of time actually it reduce the average handle Time by lot”transcript, stated

Alternatives considered (roads not taken)

Replace the banking chatbot with a fully generative bot

Rejected because: Scotiabank used LLMs first for summarization and sustainment around the existing controlled chatbot rather than letting generation own every customer response.

Adding training phrases without impact analysis

Rejected because: Peach scores similarity against existing training phrases to reduce unintended model effects.

Manual two-trainer sustainment cycle

Rejected because: Luigi, Eva, and Peach automate parts of the repeated utterance-review and intent-improvement loop.

Patterns

data-quality-loop (data)

The sustainment system uses customer query data to identify misclassifications, propose intents, and evaluate training phrase additions.

confidence-thresholding (evaluation)

Peach gives similarity scores so trainers can judge whether adding a phrase will improve or harm intent training data.

prompt-chaining (orchestration)

The generative AI summarizer went through a deliberate prompt-engineering process before producing handoff summaries.

safety-policy-enforcement (safety)

For specific sensitive fields like transactions and amounts, the team constrained extraction so the LLM could not freely hallucinate those details.

Tradeoffs

How to automate chatbot conversations in banking.

Chose intent-based chatbot plus targeted ML/LLM support over fully open-ended LLM chatbot.

Rationale: The system keeps the customer-facing bot controlled while using ML and LLMs for sustainment and handoff summarization.

Where to use generative AI first.

Chose conversation summarization for live-agent handoff over generating customer-facing banking answers directly.

Rationale: Summaries help agents work faster while keeping final customer handling with a human advisor.

Gotchas (what bit them)

Chatbot quality maintenance required repeated human review.

Root cause: Customer queries continually reveal coverage and accuracy gaps in intent classification.

Resolution: Use Luigi, Eva, and Peach to automate parts of the query review and training-data improvement cycle.

Adding training phrases can have unintended effects.

Root cause: New phrases alter the intent training set and can shift classification behavior.

Resolution: Use a similarity algorithm to preview how a phrase relates to existing examples before adding it.

LLM summarization can hallucinate banking transaction details.

Root cause: Generative summaries may invent or alter specific values if those values are left unconstrained.

Resolution: Use parameters to extract and protect specific fields such as transaction and amount information.

Evaluation

Scotiabank evaluated the sustainment models through their ongoing chatbot training cycle and measured the summarizer by reduction in conversation length and live-agent handle-time impact.

Metrics: over 700 chatbot response types · almost 1,000 hours saved annually · conversation summaries reduce conversation length by 80% · reduced average handle time

Outcomes

• Three ML models automated parts of Scotiabank's chatbot sustainment cycle.
• The summarizer reduced chatbot conversations by about 80% before live-agent handoff.
• The LLM summarization feature became the bank's first production generative AI use case according to the project summary.

The transcript is an auto-captioned conference talk; exact wording is noisy, but the architecture and measures are clear across summary and transcript.