Building an AI Private Banker with Agentic Systems for Customer Service and Financial Operations

nubank · Finance · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Digital banking customer service and financial operations

Nubank wanted to make AI act like a private banker for a very large customer base, handling customer-service contacts and money-transfer flows while preserving accuracy, trust, and regulatory-grade reliability.

Constraints:

• Financial actions require very low inaccuracy because every dollar matters.
• Customer service runs at millions of contacts per month.
• Money transfer has to work across voice, image, and chat.
• Quality evaluation needs to scale beyond manual review.

Architecture

Nubank built two major LLM applications: customer-service agents that first handle a large share of chat contacts, and an agentic money-transfer system that can process voice, image, and chat instructions. The team uses tracing and iterative LLM-as-judge evaluation, moving through prompt and fine-tuning iterations until automated judgment neared human evaluator performance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer[Customer] --> customer-service-agent[Customer Service Agent]
  customer[Customer] --> money-transfer-agent[Money Transfer Agent]
  money-transfer-agent --> authentication-and-confirmation[Authentication and Confirmation]
  money-transfer-agent --> banking-systems[Banking Systems]
  customer-service-agent --> tracing-and-evaluation[Tracing and Evaluation]
  money-transfer-agent --> tracing-and-evaluation[Tracing and Evaluation]

Components

Data flow

Customer chat contacts are first handled by LLMs. For money transfer, a user connects through WhatsApp, authenticates, gives a transfer instruction, confirms the recipient and amount, and the system executes the transfer. Evaluation traces feed iterative prompt and fine-tuning work.

Why this architecture

Nubank separates customer-service automation from high-stakes money movement but evaluates both through production tracing. Confirmation steps and low inaccuracy targets preserve trust for financial actions, while LLM-as-judge iteration scales quality measurement.

Models used: LLMs · fine-tuned evaluator model

Scale

• Nubank had almost 120 million users.
  “no surprise um we have 120 million almost users Right. And we get almost 8 and a half million um contacts every month. And out of that chat is our main um channel and in that channel right now 60% of them are first dealt with LLMs.”transcript, stated
• Nubank received almost 8.5 million contacts per month.
  “almost users Right. And we get almost 8 and a half million um contacts every month. And out of that chat is our main um channel and in that channel right now 60% of them are first dealt with LLMs.”transcript, stated
• LLMs first handled 60% of chat contacts.
  “um channel and in that channel right now 60% of them are first dealt with LLMs. the results are uh improving and we are building agents for different situations and I'll talk about them and the next one is actually more interesting and”transcript, stated
• The money-transfer flow dropped from around 70 seconds through nine screens to less than 30 seconds.
  “earlier it used to take around 70 seconds to make this transfer through nine different screens it's taking less than 30 seconds now”transcript, stated
• The money-transfer agent reported over 90% CSAT and less than 0.5% inaccuracy.
  “you can see the seesat is more than 90% uh less than 0.5% inaccuracy so on and so forth and we are doing that at scale and I will talk about that where evaluations matter and what kind of things matter so just to be on the same”transcript, stated

Alternatives considered (roads not taken)

Traditional nine-screen transfer flow.

Rejected because: The agentic flow made transfers faster while preserving confirmation.

Manual-only evaluation.

Rejected because: Automated judging reached near-human F1 and enabled fast iteration.

Single prompt/model attempt.

Rejected because: The system improved through six iterations of fine-tuning and prompt changes.

Patterns

human-in-the-loop-gate (financial-safety)

The transfer flow asks for explicit confirmation before executing money movement.

llm-as-judge (evaluation)

Nubank iterated an evaluator until automated F1 was comparable with humans.

observability-tracing (llmops)

Online tracing enabled rapid prompt and fine-tuning iterations.

Tradeoffs

Transfer UX

Chose Agentic voice/image/chat flow with confirmation over Nine-screen manual flow.

Rationale: The new flow is faster but still confirms before executing.

Evaluator iteration

Chose Fine-tuned and prompt-iterated LLM judge over Static evaluator.

Rationale: Six iterations improved F1 from 51 to 79 and better caught inaccuracies.

Gotchas (what bit them)

Financial agents need a much tighter error budget than ordinary chatbots.

Root cause: Money movement creates direct customer and trust impact.

Resolution: Use confirmation and measure inaccuracy separately from satisfaction.

Evaluation F1 can hide which error type matters more.

Root cause: A lower overall F1 may catch more inaccuracies.

Resolution: Choose the iteration that better identifies inaccurate information, not just the highest score.

Evaluation

Nubank evaluates with CSAT, inaccuracy rate, transfer time, F1 against human judgment, and online traces from real LLM interactions.

Metrics: CSAT · Inaccuracy rate · Transfer time · F1 score · Human-comparison F1 · Monthly contact volume

Outcomes

• LLMs first handled 60% of customer-service chat contacts.
• Money transfer time fell from around 70 seconds to under 30 seconds.
• The LLM judge reached 79% F1 compared with 80% for humans.

The source reports 120 million users, 8.5 million monthly contacts, 60% LLM-first chat handling, sub-30-second transfer, over-90% CSAT, under-0.5% inaccuracy, and six evaluation iterations.