Multimodal RAG Architecture Optimization for Production

microsoft · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise multimodal RAG

Microsoft ISE needed a RAG pipeline that could answer enterprise document questions grounded in both text and images, while making ingestion and retrieval efficient enough for production use.

Constraints:

• Enterprise documents contain both text and visual content.
• Image enrichment adds ingestion latency and cost.
• Retrieval quality depends on chunk structure, metadata, classifier thresholds, and model choice.
• Inference and ingestion models had different performance profiles.

Architecture

The team enriched document images with GPT-4V descriptions at ingestion, stored those descriptions with text in Azure AI Search, and systematically tuned chunking, prompts, classifiers, metadata, and inference model selection.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  document-ingestion["Document Ingestion"]
  image-filtering["Image Filtering"]
  gpt-4v-image-enrichment["GPT-4V Image Enrichment"]
  text-chunking["Text Chunking"]
  azure-ai-search["Azure AI Search"]
  inference-model["Inference Model"]
  document-ingestion --> image-filtering
  document-ingestion --> text-chunking
  image-filtering --> gpt-4v-image-enrichment
  gpt-4v-image-enrichment --> azure-ai-search
  text-chunking --> azure-ai-search
  azure-ai-search --> inference-model
  inference-model --> azure-ai-search

Components

Data flow

Documents are ingested, image-containing pages are filtered, GPT-4V generates descriptions for selected images, text and image chunks plus metadata are indexed in Azure AI Search, and the inference model retrieves grounded context to answer user queries.

Why this architecture

Separating image enrichment from inference let the team use the strongest model for each stage while indexing image meaning in the same retrieval system as text.

Models used: GPT-4V · GPT-4o · Azure AI Search

Scale

• Image filtering reduced ingestion latency by 30-50%.
  “reducing ingestion latency by 30-50% through intelligent image filtering.”summary, stated

Alternatives considered (roads not taken)

Inline image descriptions inside text chunks

Rejected because: Separate image chunks performed better for retrieval.

Use the same model for image enrichment and inference

Rejected because: GPT-4V was better for enrichment while GPT-4o was better for inference.

Patterns

rag-retrieval (retrieval)

Text and generated image descriptions are indexed together for grounded multimodal retrieval.

eval-driven-development (evaluation)

The team used systematic experiments to choose prompts, chunking, thresholds, metadata, and models.

model-routing (model-selection)

Different models were selected for enrichment and inference based on observed performance.

Tradeoffs

Filter images before enrichment

Chose Classifier-thresholded enrichment over Enrich every image in every document.

Rationale: Filtering preserved retrieval improvement while reducing ingestion latency.

Use document metadata in retrieval

Chose Metadata-aware source retrieval over Content-only retrieval.

Rationale: Metadata improved source recall.

Gotchas (what bit them)

Image enrichment can make ingestion too slow if applied indiscriminately.

Root cause: Every image sent to a vision model adds processing cost and latency.

Resolution: Use intelligent image filtering before enrichment.

Naive chunk representation can bury image evidence.

Root cause: Inline descriptions did not retrieve as well as separate image chunks.

Resolution: Index image descriptions as separate retrievable chunks.

Evaluation

The team ran controlled experiments over pipeline components and measured retrieval metrics plus ingestion latency.

Metrics: Retrieval metrics · Source recall · Ingestion latency

Outcomes

• Image filtering reduced ingestion latency by 30-50%.
• Separate image chunks improved retrieval over inline descriptions.
• Metadata improved source recall.

The report gives a 30-50% ingestion-latency reduction but does not disclose corpus size or production traffic.