Building Production-Grade RAG Systems for Financial Document Analysis

microsoft · Finance · 2023 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial document RAG over analyst reports, SEC filings, tables, charts, and graphs

Microsoft development teams needed a production-grade RAG system for financial document analysis that could retrieve precise evidence from tens of thousands of complex documents, handle metadata and fiscal periods correctly, process tables/charts/graphs, avoid lost-in-the-middle failures, and support safe iteration through robust evaluation.

Constraints:

• The corpus includes tens of thousands of financial documents, each potentially hundreds of pages long.
• Large context windows do not remove the need for retrieval because cost, latency, throughput, and lost-in-the-middle behavior make brute-force context stuffing unreliable.
• Similarity search alone can retrieve the wrong SEC filing unless document metadata such as company, filing period, fiscal year, ticker, and date are extracted and applied.
• Financial visuals may contain answer-critical information that does not appear in text, but processing every image with a multimodal model is expensive and slow.
• Prompt and processing changes can dramatically alter accuracy while looking harmless in code review, so evaluation must be built from day one.

Architecture

The system uses a two-part RAG architecture: a data-ingestion pipeline and a generation pipeline. Ingestion parses financial documents with Azure Document Intelligence, separates text/tables/graphical elements, extracts metadata, converts dates to fiscal periods, classifies images to filter out irrelevant visuals, describes relevant charts and graphs with multimodal models, stores chart descriptions as JSON, keeps original images for complex cases, embeds content, and writes it into a vector database. The generation pipeline filters retrieval by metadata, retrieves relevant chunks or visual descriptions, optionally invokes a multimodal model on original images, and generates answers. An evaluation harness compares system versions using claim decomposition, numerical tolerance, domain-specific calibration, and segment-level evaluation across tables, charts, text, and document types.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  financial-document-corpus["Financial document corpus"] --> data-ingestion-pipeline["Data ingestion pipeline"]
  data-ingestion-pipeline --> azure-document-intelligence-parser["Azure Document Intelligence parser"]
  azure-document-intelligence-parser --> metadata-extractor["Metadata extractor"]
  azure-document-intelligence-parser --> image-classifier["Image classifier"]
  azure-document-intelligence-parser --> vector-database["Vector database"]
  metadata-extractor --> fiscal-period-converter["Fiscal period converter"]
  metadata-extractor --> retriever-with-filters["Retriever with filters"]
  fiscal-period-converter --> evaluation-harness["Evaluation harness"]
  image-classifier --> multimodal-chart-describer["Multimodal chart describer"]
  multimodal-chart-describer --> chart-json-store["Chart JSON store"]
  multimodal-chart-describer --> vector-database
  chart-json-store --> vector-database
  original-image-store["Original image store"] --> generation-pipeline["Generation pipeline"]
  vector-database --> retriever-with-filters
  retriever-with-filters --> answer-generator["Answer generator"]
  generation-pipeline --> retriever-with-filters
  generation-pipeline --> original-image-store
  generation-pipeline --> answer-generator
  answer-generator --> evaluation-harness
  evaluation-harness --> claim-decomposition-evaluator["Claim decomposition evaluator"]
  evaluation-harness --> numerical-tolerance-evaluator["Numerical tolerance evaluator"]
  evaluation-harness --> segmented-eval-reports["Segmented eval reports"]

Components

Data flow

Financial documents enter the ingestion pipeline. Azure Document Intelligence parses pages into text, tables, and graphical elements. Metadata extraction captures titles, dates, company names, stock tickers, reporting periods, and fiscal calendars. Text/table chunks and chart descriptions are embedded and stored in the vector database; original images are retained for direct multimodal analysis when chart descriptions are insufficient. At query time, metadata filters narrow candidate documents, similarity retrieval fetches relevant chunks/descriptions, and the generator produces an answer. Evaluation compares generated answers against expert answers by decomposing claims, checking missing claims and contradictions, applying numerical tolerance, and segmenting performance by content type.

Why this architecture

The architecture exists because production financial RAG is not solved by large context windows or simple similarity search. Cost, latency, throughput, lost-in-the-middle behavior, duplicated filings, fiscal-calendar quirks, visual-only answers, and fragile prompt behavior require precise ingestion, metadata filters, selective multimodal processing, and evaluation-driven development.

Models used: Azure Document Intelligence · Embedding models · Multimodal chart/graph description model · Multimodal generation-time visual analysis model · LLM answer generator · LLM-as-judge evaluation prompts · Image classifier

Scale

• The system targets tens of thousands of financial documents.
  “tens of thousands of documents, each potentially containing hundreds of pages”summary, stated
• GPT-4 is described as supporting 128K tokens.
  “GPT-4 supports 128K tokens”summary, stated
• Gemini 1.5 Pro is described as supporting 2 million tokens, about 4,000 pages.
  “Gemini 1.5 Pro can handle 2 million tokens (approximately 4,000 pages).”summary, stated
• The lost-in-the-middle experiment used a 212-page document of about 118K tokens.
  “using a 212-page document (approximately 118K tokens)”summary, stated
• The same answer sentence worked at page 5 but failed at page 212.
  “When the information was at page 5, the model correctly identified”summary, stated

  “When the same information was at page 212, the model claimed it had no knowledge”summary, stated
• The evaluation example compares $4.7 million against approximately $5 million.
  “Given an expert answer stating "4.7 million dollars"”summary, stated

  “the system returned "approximately 5 million dollars"”summary, stated

Alternatives considered (roads not taken)

Dump all documents into a large context window.

Rejected because: The team cites cost, latency, throughput, and lost-in-the-middle failures as reasons precise retrieval remains necessary even with very large context windows.

Use similarity search without metadata filtering.

Rejected because: Financial corpora contain many similar filings, and retrieval can return the wrong company or time period without metadata filters.

Describe every extracted image with a multimodal model.

Rejected because: The corpus included many irrelevant photos, decorative elements, logos, and converted headers; processing all of them was too costly and slow.

Rely only on generated chart descriptions.

Rejected because: Complex visualizations with many data points or overlapping elements may require storing originals and invoking a multimodal model during generation.

Use binary correct/incorrect evaluation only.

Rejected because: Binary evaluation is too coarse for financial answers where approximations, missing reasons, and material numerical differences need separate treatment.

Patterns

rag-retrieval (architecture)

The system retrieves relevant chunks from a vector database instead of putting entire filings and reports into the model context.

retrieval-ranking (architecture)

Retrieval is constrained by extracted metadata such as company, filing period, fiscal year, date, and ticker so similar filings do not swamp the result set.

model-routing (architecture)

The image classifier routes only informational visuals to expensive multimodal chart processing and keeps irrelevant images out of that path.

cost-aware-routing (operations)

The pipeline avoids sending irrelevant images and overly large contexts to expensive models unless the content needs that processing.

eval-driven-development (quality)

Evaluation is built from day one, segmented by content type, calibrated for financial numerical tolerance, and used to compare system versions safely.

llm-as-judge (evaluation)

The evaluation prompt decomposes expert and model answers into claims, missing claims, contradictions, and material numerical differences.

structured-output-contract (architecture)

Chart/graph descriptions are stored as JSON objects so they can be indexed and retrieved.

Tradeoffs

Use RAG instead of brute-force large-context prompting.

Chose Precise retrieval with metadata filters over Sending thousands of pages into the context window.

Rationale: RAG controls cost, latency, throughput, and lost-in-the-middle behavior.

Use metadata-filtered retrieval for financial filings.

Chose Similarity retrieval plus document metadata over Similarity-only retrieval.

Rationale: Similarity-only retrieval can select the wrong quarter or fiscal year among nearly identical filings.

Describe simple charts at ingestion but keep originals for complex visuals.

Chose JSON descriptions plus original image fallback over Only one visual representation strategy.

Rationale: Descriptions make simple visuals searchable, while complex visuals may need direct multimodal analysis despite added cost and latency.

Filter images before multimodal processing.

Chose A simple informational-image classifier over Sending every extracted image to multimodal processing.

Rationale: Most real-world images were not useful charts/graphs, and multimodal processing them all was costly.

Use calibrated claim-level evaluation.

Chose Claim decomposition plus numerical tolerance over Binary correct/incorrect grading.

Rationale: Financial answers often need to distinguish missing reasons, contradictions, and acceptable rounding.

Gotchas (what bit them)

Large context windows still lose relevant financial facts.

Root cause: The model may fail to retrieve facts located late or in the middle of a long context, even when the information is present.

Resolution: Use precise retrieval rather than brute-force context stuffing.

Financial-document similarity can retrieve the right topic from the wrong filing period.

Root cause: Dozens of SEC filings from the same company can contain nearly identical language across quarters.

Resolution: Extract company, filing period, fiscal year, title, date, and ticker metadata and apply it as retrieval filters.

Fiscal-period prompts can be fragile.

Root cause: Small wording changes, such as asking for only the final answer instead of reasoning, can break company-specific fiscal-calendar conversion.

Resolution: Evaluate prompt changes against fiscal-year conversion cases before accepting them.

Charts and graphs contain answer-critical data absent from text.

Root cause: Financial documents often encode facts in pie charts, graphs, tables, diagrams, and other visual elements.

Resolution: Describe visual elements during ingestion and store original images for direct multimodal analysis when necessary.

Multimodal processing cost explodes on real document corpora.

Root cause: PDF conversions and reports produce many non-informational images such as photos, logos, decorations, and headers.

Resolution: Add an image classifier before multimodal processing to filter irrelevant visuals.

Binary correctness evaluation is too coarse for financial answers.

Root cause: Answers can be partially correct, miss one reason, or differ numerically in a way that may or may not be material.

Resolution: Use claim decomposition and domain-specific numerical tolerance.

Evaluation

The team advocates automated, domain-calibrated evaluation from day one. Their progression starts with binary correctness, then claim decomposition to identify missing claims and contradictions, then numerical tolerance for financial statements. Evaluation is segmented by tables, charts, text, and document types to compare system versions safely.

Metrics: Binary correctness · Claim coverage · Missing claims · Contradictions · Numerical tolerance/materiality · Performance by tables · Performance by charts · Performance by text · Performance by document type · System-version comparison confidence · Human-curated evaluation quality

Outcomes

• The team improved the ingestion side of the RAG architecture relative to a baseline and focused on metadata, visual processing, and evaluation.
• Lost-in-the-middle experiments justified retrieval over context stuffing for long financial documents.
• Metadata extraction and fiscal-period handling addressed wrong-filing retrieval risk.
• Image classification reduced unnecessary multimodal processing on irrelevant visuals.
• Evaluation methodology became the mechanism for safe system-version comparison and iterative improvement.

The summary reports tens of thousands of documents, hundreds of pages per document, GPT-4's 128K context, Gemini 1.5 Pro's 2M-token/4,000-page context, a 212-page/118K-token lost-in-the-middle experiment, and a financial evaluation example comparing $4.7M to approximately $5M.