Building a Gradual, Trust-Focused GenBI Agent for Enterprise Data Democratization

northwestern_mutual · Finance · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise generative business intelligence

Northwestern Mutual wanted a GenBI agent that lets business users answer questions with enterprise data while reducing BI-team dependency, but the company had to manage accuracy, trust, governance, risk aversion, messy real data, and leadership buy-in.

Constraints:

• the company is a 160-year-old financial services, life insurance, and wealth-management firm with strong stability expectations
• real enterprise data is messy and hard for a BI agent to understand
• users and leaders needed trust before broad rollout
• executives require clear, concise, trustworthy answers and were not an early target
• SQL generation was considered too risky and complex as the first release path

Architecture

Northwestern Mutual built a gradual GenBI architecture with a data/metadata layer, orchestration, governance/trust controls, contextual UI, and four specialized agents: metadata, RAG, SQL, and BI. The first productized path retrieves certified reports rather than generating SQL. If a certified report is insufficient, the SQL agent uses report context as a close seed for a query, and the BI agent translates data into a business answer. The rollout follows a crawl-walk-run trust model: BI experts first, then business managers, executives later.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  business-question-interface[Business question interface] --> orchestrator[Orchestrator]
  orchestrator --> metadata-agent[Metadata agent]
  data-and-metadata-layer[Data and metadata layer] --> metadata-agent
  metadata-agent --> rag-agent[RAG agent]
  rag-agent --> certified-reports[Certified reports]
  rag-agent --> sql-agent[SQL agent]
  data-and-metadata-layer --> sql-agent
  certified-reports --> bi-agent[BI agent]
  sql-agent --> bi-agent
  bi-agent --> business-question-interface
  governance-and-trust-layer[Governance and trust layer] --> orchestrator
  governance-and-trust-layer --> certified-reports

Components

Data flow

A business question enters the orchestrator. The metadata agent uses the catalog and documentation to understand the question context. The RAG agent searches certified reports and dashboards. If no suitable report exists, the SQL agent creates or expands a query using the closest report/query as a seed. The BI agent converts the retrieved or queried data into a business answer under governance, trust, and orchestration controls.

Why this architecture

The architecture intentionally starts with trusted assets rather than open-ended SQL generation. Certified report retrieval delivers immediate BI-team capacity savings while keeping answers aligned with approved data. Metadata enrichment and agent components are productized incrementally so leadership can see value and stop or continue after each phase.

Models used: LLMs for metadata, RAG, SQL, and BI agents

Scale

• Northwestern Mutual is a 160-year-old financial services, life insurance, and wealth management company.
  “work. So we're a financial services, life insurance, and wealth management. Been around for 160 years. [snorts] Uh, some very impressive numbers there. But first of all, I want to say why is Northwestern Mutual a great place to do Gen AI. We got a lot of data, we got a”transcript, stated
• About 20 percent of BI-team capacity involved finding and sending the right report.
  “the rag agent that pulls the right report uh allowed us to take about 20% of the overall capacity of the BI team”transcript, stated
• The RAG report-finding agent automated around 80 percent of that 20 percent capacity.
  “person. So we were able to automate around 80% out of those uh 20% and we're talking about a team of 10 people. So roughly two people full-time job all they do is find the right report and send it to the right person.”transcript, stated
• The report-finding work corresponded to roughly two full-time people on a 10-person team.
  “around 80% out of those uh 20% and we're talking about a team of 10 people. So roughly two people full-time job all they do is find the right report and send it to the right person.”transcript, stated

Alternatives considered (roads not taken)

Start with SQL generation

Rejected because: SQL generation was too complex and risky for the initial trust-building phase.

Roll out directly to executives

Rejected because: Executives need concise, trusted answers and the system was not accurate enough for that stage.

Use synthetic or clean demo data

Rejected because: The team wanted real messy data because that is where enterprise challenges appear.

Buy third-party GenBI from outside

Rejected because: The source says third-party solutions are harder to govern from the outside, though the decision remains TBD.

Patterns

multi-agent-orchestration (orchestration)

The GenBI pipeline uses specialized metadata, RAG, SQL, and BI agents coordinated by an orchestrator.

rag-retrieval (retrieval)

The first production value comes from retrieving certified reports and dashboards before generating new queries.

human-in-the-loop-gate (workflow)

Rollout starts with BI experts who can judge correctness and provide feedback before broader business users.

confidence-thresholding (risk)

The crawl-walk-run rollout limits exposure based on how much users can recognize incorrect output.

data-quality-loop (data)

Metadata experiments proved that enriched metadata measurably improved LLM performance, justifying further documentation work.

Tradeoffs

Use a crawl-walk-run rollout

Chose BI experts first, then business managers, executives later over broad democratization from day one.

Rationale: Early users needed enough expertise to recognize mistakes and give useful feedback.

Deliver certified reports first

Chose RAG over trusted reports and dashboards over generating new SQL-backed answers immediately.

Rationale: Certified reports were already trusted and covered a large share of BI-team support work.

Productize components incrementally

Chose separate productizable metadata, RAG, SQL, and BI pieces over waiting for a full end-to-end GenBI agent before delivering value.

Rationale: Each phase could show tangible impact and let leadership stop or fund the next step.

Gotchas (what bit them)

Trust has to be earned separately with users and leadership.

Root cause: The company is risk-averse and GenBI accuracy barriers are visible to everyone.

Resolution: Use incremental deliverables, BI-expert rollout, certified reports, and leadership-visible business value.

Real enterprise data is messy and different from demos.

Root cause: A 160-year-old company has actual messy data and enterprise-specific metadata/context needs.

Resolution: Use real data from the start and learn what good metadata looks like for a BI agent.

Open-ended SQL generation is risky as a first milestone.

Root cause: SQL is difficult even for people and can produce untrusted answers.

Resolution: Start by retrieving existing certified reports, then use SQL only when report context is insufficient.

Evaluation

Northwestern Mutual evaluated value through incremental business deliverables, automation of BI report-finding capacity, and A/B-style metadata tests comparing LLM performance with and without enriched metadata.

Metrics: 20% of BI-team capacity tied to finding/sharing reports · 80% of that 20% automated · team of 10 people with roughly two full-time equivalents on report finding · battery of questions against databases with and without good metadata

Outcomes

• the RAG report agent automated a large portion of BI report-finding work
• metadata experiments proved value back to senior leadership
• each agent/component could be delivered as a separate product with business value
• the project earned continued quarter-over-quarter leadership investment

The source gives organizational rather than traffic scale: 160-year company, a 10-person BI team slice, 20% capacity affected, and 80% automation of that slice.