AskNu: RAG-Based Employee Knowledge Management System

nubank · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Employee knowledge management

Nubank needed employees to find accurate internal documentation quickly across autonomous Confluence spaces without forcing every question into a support ticket.

Constraints:

• around 9,000 employees work across multiple teams and global locations
• each team owns its own Confluence processes and documentation
• documentation changes frequently, making repeated fine-tuning too costly
• answers must avoid mixing documentation from different departments
• users need citations and a fallback path when the answer is unsatisfactory

Architecture

AskNu is a Slack-integrated RAG system over Nubank’s Confluence documentation. An automated indexing job extracts text from every department page, chunks documents with metadata, embeds them, and refreshes the knowledge base every two hours. At query time, AskNu performs a first search to support dynamic few-shot department classification, then performs a second search only inside the classified department and feeds those chunks to an LLM to generate a personalized answer with document URLs. If the answer is not satisfactory, the user is routed to the relevant department portal to open a ticket.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  employee-slack-client["employee-slack-client"] --> asknu-rag-service["asknu-rag-service"]
  confluence-documentation["confluence-documentation"] --> confluence-indexer["confluence-indexer"]
  confluence-indexer["confluence-indexer"] --> embedding-model["embedding-model"]
  embedding-model["embedding-model"] --> knowledge-base["knowledge-base"]
  asknu-rag-service["asknu-rag-service"] --> first-pass-retriever["first-pass-retriever"]
  first-pass-retriever["first-pass-retriever"] --> knowledge-base["knowledge-base"]
  first-pass-retriever["first-pass-retriever"] --> dynamic-few-shot-classifier["dynamic-few-shot-classifier"]
  dynamic-few-shot-classifier["dynamic-few-shot-classifier"] --> department-scoped-retriever["department-scoped-retriever"]
  department-scoped-retriever["department-scoped-retriever"] --> knowledge-base["knowledge-base"]
  department-scoped-retriever["department-scoped-retriever"] --> answer-generation-llm["answer-generation-llm"]
  answer-generation-llm["answer-generation-llm"] --> asknu-rag-service["asknu-rag-service"]
  asknu-rag-service["asknu-rag-service"] --> department-ticket-portal["department-ticket-portal"]

Components

Data flow

Confluence department pages are periodically extracted, chunked, embedded, and stored with document metadata. A Slack user asks AskNu a question. The first retrieval pass supplies candidate chunks and departments to a dynamic few-shot classifier that chooses a department or External Source. AskNu then searches only the selected department’s documents, sends the retrieved chunks and user query to the LLM, returns a personalized answer with URLs, and redirects dissatisfied users to the appropriate ticket portal.

1. For the routing: the retrieved chunks are fed into a LLM to perform a Few-Shot classification, in a process we call Dynamic Few-Shot Classification."

Why this architecture

Nubank chose RAG because internal documentation changes too often for repeated fine-tuning to be economical. The two-stage retrieval design prevents the answer model from mixing documents across departments, while Slack integration and URL references keep the tool inside employee workflows and auditable. The two-hour refresh cadence keeps team-owned documentation current without removing autonomy from the teams that own it.

Models used: Large Language Model · embeddings · dynamic few-shot classification · RAG

Scale

• Nubank had around 9,000 employees.
  “With around 9,000 employees and multiple teams across the globe”article, stated
• The index refreshes every two hours.
  “this process is automatically triggered every 2 hours”article, stated
• After six months of general availability, AskNu reported 5,000 active users and 280,000 messages.
  “6 months after the tool has been generally available for all employees”article, stated

  “After six months of deployment, the system achieved 5,000 active users, processed 280,000 messages, received 80% positive feedback, reduced support tickets by 96%, and decreased information retrieval time from 30 minutes (or up to 8 hours with tickets) down to 9 seconds.”summary, stated
• AskNu returned answers in 9 seconds versus up to 30 minutes on Confluence or 8 hours with a ticket.
  “9 seconds to get an answer on AskNu (the time to find the answer on Confluence takes up to 30 minutes or 8 hours”article, stated
• Internal-domain answer accuracy was audited at 74%.
  “74% of internal domain requests answers have been labeled as accurate.”article, stated

Alternatives considered (roads not taken)

Fine-tune the LLM whenever documentation changes

Rejected because: New information appears constantly, making repeated fine-tuning too costly.

Search the whole documentation corpus for final answer generation

Rejected because: Whole-corpus retrieval risked mixing department documentation and producing inaccurate or confusing answers.

Force every failed answer into manual search

Rejected because: AskNu keeps a structured fallback to the department portal so support tickets are still routed correctly.

Patterns

rag-retrieval (retrieval)

AskNu retrieves current Confluence chunks in real time and feeds them to an LLM for answer generation.

model-routing (orchestration)

A dynamic few-shot classifier routes each question to the right department or External Source before answer retrieval.

retrieval-ranking (retrieval)

The first retrieval step supplies the few-shot examples and department candidates used for classification.

human-in-the-loop-gate (fallback)

Unsatisfied users are redirected to the relevant department support portal.

data-quality-loop (governance)

Department owners audit answers and use answer quality to find documentation improvement opportunities.

Tradeoffs

Use RAG instead of repeated fine-tuning

Chose real-time retrieval from Confluence over fine-tuning whenever documentation changes.

Rationale: RAG gives fresher answers at lower operational cost for dynamic internal documentation.

Use two-stage retrieval

Chose department classification followed by scoped search over single-stage whole-corpus answer retrieval.

Rationale: The first stage reduces cross-domain confusion; the second stage generates the answer from a narrower and more relevant document set.

Gotchas (what bit them)

RAG over all Confluence spaces can mix unrelated departmental policies.

Root cause: Autonomous teams own separate documentation domains with overlapping language.

Resolution: Use a first-pass department classifier, then retrieve only from that department for final answer generation.

Documentation freshness makes fine-tuning an operational mismatch.

Root cause: Relevant internal information can be created at any time.

Resolution: Refresh the RAG knowledge base every two hours instead of repeatedly fine-tuning the model.

Evaluation

AskNu measures user feedback, response rate, answer accuracy through department-owner audits, answer latency, and ticket-reduction outcomes after general availability.

Metrics: 80% positive feedback · 6% response rate · 74% internal-domain answer accuracy · 9 seconds to answer · 30 minutes Confluence search baseline · 8 hours ticket baseline · support-ticket reduction

Outcomes

• AskNu reached broad internal use after six months
• employees received answers in about 9 seconds
• positive feedback reached 80% among respondents
• internal-domain answer accuracy was labeled 74%
• department owners gained a feedback loop for documentation quality

The source reports a 9,000-employee organization, two-hour knowledge-base refresh, 5,000 active users, 280,000 messages, and answer retrieval time reduced to 9 seconds.