Building a Client-Focused Financial Services Platform with RAG and Foundation Models

mnp · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial services GenAI platform

MNP needed a client-focused GenAI platform that could provide contextualized insights while overcoming tight foundation-model/data-warehouse coupling, laggy time-to-first-token behavior, GPU-driven cost, and security requirements for diverse client datasets.

Constraints:

• foundation-model integration with the existing data warehouse limited experimentation
• time-to-first-token lag hurt user experience
• large data volume and frequent retraining made GPU TCO prohibitive
• client data must remain isolated and secure
• datasets need regular updates and embedding refreshes

Architecture

MNP partnered with Databricks to build a lakehouse-based RAG platform. Databricks AI Search provides governed vector search, Mixtral 8x7B serves as the selected mixture-of-experts model, Foundation Model APIs connect lakehouse compute to the application, RAG keeps responses grounded in continually refreshed data, and Unity Catalog enforces Private AI standards and governance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  databricks-lakehouse[Databricks lakehouse] --> ai-search-vector-database[AI Search vector database]
  databricks-lakehouse --> unity-catalog[Unity Catalog]
  ai-search-vector-database --> rag-retrieval-layer[RAG retrieval layer]
  rag-retrieval-layer --> mixtral-8x7b-model[Mixtral 8x7B model]
  mixtral-8x7b-model --> foundation-model-apis[Foundation Model APIs]
  foundation-model-apis --> client-facing-interface[Client-facing interface]
  unity-catalog --> client-facing-interface

Components

Data flow

Client and firm data lands in the Databricks lakehouse. AI Search stores embeddings in a governed serverless vector database, retrieval brings relevant context into the generation flow, Mixtral 8x7B produces the response through Foundation Model APIs, and Unity Catalog maintains governance and data isolation.

Why this architecture

The lakehouse plus RAG architecture decoupled foundation-model experimentation from the old warehouse, supported updated data through refreshed embeddings, and kept client data isolated under governance controls. Mixtral 8x7B's MoE design fit simultaneous workloads and context-specific sensitivity.

Models used: Mixtral 8x7B · DBRX · foundation models

Scale

• The data team built a model from start to quality assurance testing within four weeks.
  “It also allowed the firm’s data team to build a model from start to quality assurance testing within four weeks.”article, stated
• MNP uses a model serving path around Mixtral 8x7B.
  “A mixture-of-experts (MoE) model like Mixtral 8x7B would enable MNP to tailor the model for context-specific sensitivity, broaden the parameters to accommodate a wider range of instructions and enhance parallel processing to support simultaneous workloads.”article, stated
• DBRX is noted as 132 billion total parameters with 36 billion active per input.
  “With 132 billion total parameters and 36 billion active parameters per input, DBRX can handle complex data tasks”article, stated

Alternatives considered (roads not taken)

Keep tightly coupled foundation-model/data-warehouse integration

Rejected because: It hindered experimentation with other model and platform choices.

Use fine-tuning/retraining as the primary refinement strategy

Rejected because: RAG was selected because datasets required regular updates or additions.

Continue GPU-heavy serving over large data volumes

Rejected because: Total cost of ownership had become prohibitive.

Patterns

rag-retrieval (retrieval)

MNP uses RAG over refreshed embeddings to provide contextual relevance from changing datasets.

privacy-preserving-processing (governance)

Unity Catalog and Private AI standards keep client data isolated and secure.

retrieval-ranking (retrieval)

AI Search acts as a governed vector repository for pinpointing relevant information.

Tradeoffs

Model refinement strategy

Chose RAG with refreshed embeddings over retraining/fine-tuning as the only path.

Rationale: The data needed regular updates and additions, making retrieval-based context more suitable.

Model choice

Chose Mixtral 8x7B over the prior tightly integrated foundation-model setup.

Rationale: The MoE model supported context-specific sensitivity, broader instructions, and parallel workloads.

Gotchas (what bit them)

Model/data-warehouse coupling blocked experimentation.

Root cause: The initial deployment tied the foundation model too tightly to existing warehouse architecture.

Resolution: Move to a lakehouse architecture with Databricks AI Search and Foundation Model APIs.

Latency and GPU costs made the initial GenAI path hard to scale.

Root cause: Time-to-first-token lag and large-data GPU usage hurt user experience and TCO.

Resolution: Adopt a more efficient lakehouse/RAG serving architecture.

Evaluation

MNP evaluated platform suitability through time-to-first-token, TCO/GPU usage, model selection, QA testing, data isolation, and production deployment readiness.

Metrics: time-to-first-token · TCO · GPU usage · four weeks to QA testing · data isolation and security

Outcomes

• MNP built a model from start to QA testing within four weeks.
• The platform supports a language-based client interface for data-driven insights.
• Unity Catalog helped maintain data isolation and security.

The case covers a national Canadian accounting, consulting, tax, and digital services firm serving diverse clients and evaluating DBRX-scale models.