Production AI Framework for Retail Banking Chatbot

databricks · Finance · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Retail banking customer support automation

A retail bank needed to turn a failed chatbot POC into a production AI agent that could deflect simple customer calls while remaining observable, measurable, governed, and diagnosable in a regulated environment.

Constraints:

• The bank had 20,000 monthly chatbot calls and wanted to automate the 60% simple-query segment.
• The prior six-month POC consumed 85K without production success.
• The system needed numeric success targets, including deflection and accuracy.
• Regulated banking required traceability, accountability, and governance around data and agent decisions.

Architecture

The Databricks framework starts with evaluation and observability before model choice. For the retail bank, the team collected 200 real support cases, defined deflection/accuracy/latency targets, built an automated evaluation pipeline with human review below thresholds, instrumented the agent flow with traces, connected APIs and a RAG vector database for policy lookup, then chose the model in week seven of an eight-week POC. The final architecture makes the agent visible, measurable, and accountable before production launch.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-banking-chatbot["Customer banking chatbot"] --> intent-classification-agent["Intent classification agent"]
  intent-classification-agent["Intent classification agent"] --> customer-account-api["Customer account API"]
  intent-classification-agent["Intent classification agent"] --> policy-retrieval-agent["Policy retrieval agent"]
  policy-retrieval-agent["Policy retrieval agent"] --> rag-vector-database["RAG vector database"]
  policy-retrieval-agent["Policy retrieval agent"] --> reasoning-step["Reasoning step"]
  reasoning-step["Reasoning step"] --> guardrail-checks["Guardrail checks"]
  guardrail-checks["Guardrail checks"] --> customer-banking-chatbot["Customer banking chatbot"]
  tracing-system["Tracing system"] --> intent-classification-agent["Intent classification agent"]
  tracing-system["Tracing system"] --> customer-account-api["Customer account API"]
  tracing-system["Tracing system"] --> policy-retrieval-agent["Policy retrieval agent"]
  tracing-system["Tracing system"] --> reasoning-step["Reasoning step"]
  tracing-system["Tracing system"] --> guardrail-checks["Guardrail checks"]
  tracing-system["Tracing system"] --> evaluation-pipeline["Evaluation pipeline"]
  evaluation-pipeline["Evaluation pipeline"] --> evaluation-dataset["Evaluation dataset"]
  evaluation-pipeline["Evaluation pipeline"] --> human-reviewer["Human reviewer"]
  model-selection-step["Model selection step"] --> evaluation-pipeline["Evaluation pipeline"]

Components

Data flow

A customer asks the banking chatbot about an overdraft fee. The agent performs intent classification, connects to the customer account through an API or database, retrieves relevant overdraft policy documents from a RAG vector database, reasons over the policy and account details, performs final guardrail checks, and responds. Traces capture each decision, timing, confidence, tool/API call, and retrieval step so failures such as stale policy documents can be diagnosed.

Why this architecture

The team deliberately inverted the normal model-first POC. Evaluation, tracing, data foundations, orchestration, and governance came before model selection so the bank could measure success, diagnose production failures, and assign accountability. The architecture reflects the speaker’s five-pillar framework: evaluation, tracing, data foundation, orchestration, and governance.

Models used: Databricks Agent Bricks · LLM judges · retail banking chatbot model

Scale

• The bank had around 20,000 customer chatbot calls per month.
  “they had got around 20,000 odd calls per month from customers on their chatbot.”transcript, stated
• Roughly 60% of the calls were simple queries targeted for agent deflection.
  “there were like 60% of them were simple queries, what is my account balance, you know, what do I do with my overdraft”transcript, stated
• The failed POC cost about 85K over six months.
  “They spent around 85K in 6 months doing a POC which did not succeed.”transcript, stated
• The rebuilt POC used 200 real support cases as its initial evaluation dataset.
  “We collected 200 cases on their actual human agents answering to their customers on simple queries and understand how they are responding to them.”transcript, stated
• The agent target was 85% accuracy and 60% simple-query handling.
  “you need 60 queries or the 60% of the queries that are simple queries to be uh to be handled by the agent, right? They needed some sort of accuracy. So, 85% it was around 85% accuracy target.”transcript, stated
• Model selection happened in week seven of an eight-week POC.
  “we selected the model in week seven, like in a eight weeks POC.”transcript, stated

Alternatives considered (roads not taken)

Model-first proof of concept

Rejected because: The prior POC failed because the team could not measure, trace, or assign accountability.

Static one-time evaluation dataset

Rejected because: The evaluation dataset is treated as a living system that grows with production failures and fixes.

Opaque chatbot responses

Rejected because: The bank needed traces showing each decision, confidence, timing, API call, retrieval, and guardrail check.

Patterns

eval-driven-development (evaluation)

Evaluation is built before model selection and defines business success numerically.

observability-tracing (operations)

The system traces every agent decision, timing, confidence, API call, retrieval, and guardrail check.

rag-retrieval (architecture)

The banking agent retrieves policy documents from a RAG vector database before reasoning.

human-in-the-loop-gate (evaluation)

Below-threshold evaluation results are checked by humans and converted into future test cases.

safety-policy-enforcement (governance)

The agent performs final guardrail checks before responding.

Tradeoffs

POC sequencing

Chose Evaluation and observability first, model in week seven over Debating and selecting models at the beginning.

Rationale: Once the dataset and metrics existed, model choice became quick and evidence-based.

Feedback handling

Chose Living evaluation dataset over One-off benchmark.

Rationale: New production failures are converted into test cases so the system catches them next time.

Answer grounding

Chose Policy-document RAG plus account API calls over Uncontextualized chatbot responses.

Rationale: The agent needs account details and current policy documents to decide whether a customer claim is legitimate.

Gotchas (what bit them)

Production failures were invisible without tracing.

Root cause: The failed POC lacked the ability to see why things failed in production.

Resolution: Instrument traces and measurement before launching.

Stale retrieval data caused wrong customer answers.

Root cause: A changed policy document was not updated in the vector database, so embeddings did not come through.

Resolution: Use feedback drops plus traces to identify outdated retrieval documents and fix the vector database.

Governance gaps create unclear accountability.

Root cause: Teams had not defined who owns failures, data assets, or customer-facing AI nonsense.

Resolution: Make governance one of the five production pillars before deployment.

Evaluation

The rebuilt POC used a 200-case evaluation dataset from real support-agent answers, numeric deflection and accuracy targets, automated response comparison, human review below thresholds, and continuous test-case growth.

Metrics: 60% simple-query deflection · 85% accuracy · latency · operational targets · CSAT/feedback · response time

Outcomes

• The bank moved from a failed six-month POC to a traced, measurable, accountable production launch.
• The system diagnosed an outdated policy-document issue after customer feedback dropped.
• Model selection was delayed until evaluation data made the choice quick.

The case reports 20,000 monthly calls, 60% simple-query deflection target, 85% accuracy target, 200 initial cases, and six-week post-launch metric review.