Responsible LLM Adoption for Fraud Detection with RAG Architecture

mastercard · Finance · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Responsible enterprise LLM adoption for fraud detection and unstructured financial data

Mastercard wanted to use LLMs and unstructured enterprise data for high-value use cases such as fraud detection, but needed to control hallucination, attribution, staleness, revision rights, access controls, scalable infrastructure, and responsible AI governance in a regulated financial environment.

Constraints:

• More than 80% of organizational data is unstructured and difficult to manage and secure.
• Enterprises need domain-specific grounding without exposing all data to a global LLM.
• LLM application code is a small part of the full production system; infrastructure, data, safeguards, and governance dominate the work.
• Financial use cases need privacy, security, reliability, access controls, and bias safeguards.
• Retrieval and generation need to work together rather than behaving like disconnected brains.

Architecture

Mastercard's responsible LLM adoption strategy centers on RAG: a foundation model is coupled to external memory from domain-specific data so answers can be grounded, attributed, updated, revised, and access-controlled. Around that core sit scalable ML infrastructure, foundation-model access, tools to customize/contextualize LLMs, data pipelines, retriever/generator optimization, specialized models for specific tasks, responsible AI principles, a governing body, and safeguards for privacy, security, reliability, access controls, and bias.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Unstructured enterprise data"] --> B["Domain-specific data pipeline"]
  A --> C["Access-control layer"]
  B --> D["Retriever and vector index"]
  C --> D
  D --> E["Foundation model generator"]
  E --> F["Specialized task models"]
  E --> G["Application layer"]
  G --> H["Scalable ML infrastructure"]
  I["Responsible AI governance"] --> C
  I --> F
  I --> G

Components

Data flow

Unstructured enterprise data is collected and secured under existing access controls. A retriever selects relevant domain-specific context or vector-index content and provides it as external memory to a foundation model/generator. The generator formulates an answer grounded in that context, with sources available for attribution and revision. Responsible-AI governance and safeguards constrain which data/model paths are available to which tasks and users, while scalable ML infrastructure supports production use.

Why this architecture

Mastercard presents RAG as the practical way to make enterprise LLMs factual, attributable, up-to-date, revisable, and customizable while preserving data access controls. The architecture acknowledges that production LLM work is mostly the surrounding system: infrastructure, data pipelines, safeguards, and governance rather than only model code.

Models used: Large Language Models · foundation models · retriever/generator RAG models

Scale

• Mastercard cites 10-15 years of prior enterprise AI value from structured data.
  “over the last 10 to 15 years, most of the AI value that we have seen is really coming from structured data.”transcript, stated
• More than 80% of organizational data is estimated to be unstructured.
  “more than 80% of the organizations data is unstructured.”transcript, stated
• About 71% of organizations struggle with managing and securing unstructured data.
  “it's also estimated that 71% of them really struggle in managing and and securing this kind of data.”transcript, stated
• Mastercard reported LLM/generative AI boosted fraud detection by up to 300% in some cases.
  “to boost fraud detection in some cases by 300%.”transcript, stated
• The talk frames ML/LLM code as less than 5% of the end-to-end production pipeline work.
  “it's less than 5% of what goes into building the end-to-end pipeline.”transcript, stated

Alternatives considered (roads not taken)

Foundation model used as-is, zero-shot/few-shot, or fine-tuned alone

Rejected because: Enterprises have accuracy, hallucination, attribution, staleness, revision, and customization constraints that standalone models do not solve cleanly.

Naive RAG with retriever and generator as separate brains

Rejected because: The speaker argues production RAG should optimize retriever and generator together, and open-source model parameters now make generator fine-tuning possible.

A global LLM system with access to all enterprise data

Rejected because: Mastercard emphasizes preserving existing access controls and using specialized models for certain tasks.

Patterns

rag-retrieval (architecture)

Mastercard uses RAG/external memory to ground LLM answers in domain-specific enterprise data and enable attribution, revision, and freshness.

safety-policy-enforcement (governance)

The adoption strategy is constrained by responsible-AI principles, a governing body, privacy/security/reliability safeguards, and bias controls.

privacy-preserving-processing (security)

Access controls from enterprise systems must be preserved in LLM/RAG ecosystems so models do not see unauthorized data.

fine-tuning-adaptation (model-adaptation)

The talk identifies fine-tuning the generator on retrieved context as a way to make RAG more factual than naive separated retriever/generator designs.

observability-tracing (operations)

RAG sources provide attribution so teams can understand why the model generated certain text or decisions.

Tradeoffs

LLM grounding strategy

Chose RAG/external memory over foundation model alone or fine-tuning alone.

Rationale: RAG addresses hallucination, attribution, staleness, revision, and customization while keeping domain data swappable and source-backed.

AI strategy focus

Chose intelligence augmentation and current-risk mitigation over speculation about AGI replacement/doomsday.

Rationale: Mastercard frames generative AI as augmenting human productivity and urges focus on risks posed by current AI systems.

Production investment

Chose surrounding LLM pipeline, safeguards, infrastructure, and governance over model/LLM code only.

Rationale: The speaker argues the surrounding system accounts for more than 90-95% of production LLM application work.

Gotchas (what bit them)

Standalone LLMs are hard to operationalize in enterprise settings.

Root cause: They create accuracy, hallucination, attribution, staleness, revision, and customization problems.

Resolution: Use RAG/external memory and surrounding production infrastructure rather than relying on the model alone.

RAG is not just attaching a separate retriever to a generator.

Root cause: Naive RAG leaves retriever and generator as disconnected brains, which may not optimize factual generation.

Resolution: Optimize or fine-tune the generator to use retrieved context, enabled by open-source model parameters.

Access control is a production RAG problem, not only an enterprise data problem.

Root cause: A global LLM/RAG system could accidentally see data beyond a user's/system's authorization.

Resolution: Preserve existing access controls and use specialized models for certain tasks.

Evaluation

The source reports a high-level production outcome for fraud detection and discusses evaluation/design concerns for LLM production systems: hallucination reduction, attribution, access controls, retriever/generator optimization, scalable infrastructure, and responsible AI safeguards.

Metrics: Up to 300% boost in fraud detection in some cases · More than 80% of organizational data is unstructured · 71% of organizations struggle managing/securing unstructured data · LLM/ML code less than 5% of the end-to-end pipeline · Surrounding LLM application boxes more than 90% of the build

Outcomes

• Mastercard reported using generative AI/LLMs to boost fraud detection by up to 300% in some cases.
• The presentation argues RAG/external memory addresses hallucination, attribution, staleness, revision, and customization problems.
• Mastercard's responsible AI strategy includes seven core principles, a governing body, safeguards, and access-control preservation.
• The talk reframes production LLM adoption as mostly infrastructure, data, safeguards, and governance work rather than model code.

The source gives one headline fraud-detection result and broad enterprise data statistics, but not a detailed benchmark or deployment architecture for the fraud model itself.