Practical Challenges in Building Production RAG Systems

prolego · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production RAG systems for enterprise documents

Prolego needed to explain why production RAG systems are much harder than tutorial examples: real client data arrives in messy formats, document hierarchy carries meaning, retrieval quality dominates answer quality, and evaluation must test faithfulness to retrieved context rather than only answer correctness.

Constraints:

• Enterprise sources include PDFs, Word documents, SharePoint sites, PowerPoint slides, wikis, emails, tables, and other messy formats.
• Document hierarchy can change the meaning of a chunk, especially for policies, exceptions, and medical or insurance records.
• Chunk size affects both embedding quality and context carried into generation.
• LLM context windows limit how many retrieved documents can be passed to the generator.
• Embedding similarity alone may retrieve outdated or irrelevant context with high-confidence citations.
• RAG evaluation must distinguish correctness from faithfulness to the retrieved evidence.
• Real users ask questions with varied vocabulary and difficulty, so happy-path examples do not characterize production performance.

Architecture

The proposed production RAG architecture parses heterogeneous enterprise documents into a flat but hierarchy-preserving representation, creates chunks whose IDs retain document location, embeds both chunks and user queries, retrieves only the most relevant material, optionally enriches retrieval with summaries, query expansion, metadata, rules, heuristics, or fine-tuned adapters, and sends retrieved context into a generative LLM. The team evaluates not only answer correctness but also whether answers are faithful to the provided context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Enterprise_document_sources["Enterprise document sources"] --> Document_parser["Document parser"]
  Document_parser --> Hierarchy_preserving_chunk_store["Hierarchy-preserving chunk store"]
  Hierarchy_preserving_chunk_store --> Embedding_model["Embedding model"]
  Query_expansion_step["Query expansion step"] --> Embedding_model
  Embedding_model --> Retrieval_system["Retrieval system"]
  Hierarchy_preserving_chunk_store --> Retrieval_system
  Summarization_step["Summarization step"] --> Retrieval_system
  Retrieval_system --> Generative_LLM["Generative LLM"]
  Generative_LLM --> Evaluation_workflow["Evaluation workflow"]
  Retrieval_system --> Evaluation_workflow
  User_guidance_layer["User guidance layer"] --> Generative_LLM

Components

Data flow

Enterprise policy, procedure, regulation, email, wiki, slide, PDF, and table sources are parsed with file-type-specific libraries and heuristics. The parser extracts elements, sorts them by page position, identifies headings and structure from fonts or colors, and stores each element in a flat list or dictionary whose key encodes hierarchy. Chunks or summaries are embedded; a user query may be rephrased several ways before embedding; similarity search, metadata filters, rules, or adapters retrieve relevant context; the generator receives only the selected context; and a separate evaluation process checks faithfulness, answer quality, and prompt difficulty.

Why this architecture

The architecture puts disproportionate effort into parsing, chunking, retrieval, and evaluation because production RAG fails when the right context is lost before generation. Preserving hierarchy keeps exceptions and qualifiers attached to their rules; retrieval experimentation is necessary because chunks that are too small or too large both degrade embeddings; query expansion and summaries improve recall under vocabulary mismatch and context limits; and faithfulness evaluation ensures the final answer is actually supported by retrieved enterprise data rather than by the model’s general knowledge.

Models used: GPT-4 · ChatGPT · embedding LLMs

Scale

• The example architecture starts with thousands of pages of employee policy documents.
  “we begin with thousands of pages of employe policy documents in emails wikis PDFs basically whatever we have”transcript, stated
• A practical PDF parsing spike may take a couple of days before useful heuristics are working.
  “you can plan on spending a couple of days maybe um you know just getting the the library up and running with some example PDFs”transcript, stated
• The source contrasts retrieving from a thousand documents with sending only relevant pieces to the LLM.
  “if we have a thousand documents we can't take all thousand documents and give them to the llm”transcript, stated
• Query expansion is described as generating several similar queries before retrieval.
  “have the llm rephrase that a couple of times or generate several similar but different queries using different words”transcript, stated
• The discussion describes generating five or six query versions for broader search in complex domains.
  “I mean generating five or six versions is going to do it broad the search”transcript, stated

Alternatives considered (roads not taken)

Naively split documents into sentences, paragraphs, or subsections

Rejected because: Chunking without hierarchy can detach exceptions, qualifiers, and negations from the rules they modify.

Flatten documents without preserving hierarchy

Rejected because: The team argues for a flat machine-learning-friendly representation whose keys still encode the original document structure.

Rely only on the best embedding model

Rejected because: The speakers treat retrieval as a larger design space involving chunking, summaries, metadata, rules, heuristics, and adapter tuning.

Evaluate only answer correctness

Rejected because: A powerful model can answer correctly from its own knowledge even when retrieval failed, so faithfulness must be evaluated separately.

Answer any user query without constraining query quality

Rejected because: The speakers note that user vocabulary and difficulty change retrieval performance, and sometimes user guidance or a reasonableness check is cheaper than six months of engineering.

Patterns

rag-retrieval (architecture)

The system retrieves organization-specific chunks and injects them into a prompt so a general LLM can answer from enterprise data.

retrieval-ranking (retrieval)

Production quality depends on ranking the right chunks, not simply computing a vector and accepting whatever comes back first.

query-rewriting (retrieval)

The team recommends asking an LLM to rephrase the user query in several ways to cover vocabulary mismatch before embedding search.

hybrid-search (retrieval)

The speakers explicitly point beyond pure embeddings to combinations with metadata, rules, and heuristics.

eval-driven-development (quality)

They frame production RAG evaluation around correctness, faithfulness, hallucination, context support, and query difficulty.

llm-as-judge (quality)

Faithfulness evaluation is described as prompting another LLM to generate and evaluate statements from question, answer, and context.

fine-tuning-adaptation (modeling)

The team mentions fine-tuning embedding models or an adapter between query embeddings and cosine similarity for application-specific matching.

data-quality-loop (operations)

The first production task is exploratory data work: get documents into the right format, preserve structure, and verify good data before modeling.

Tradeoffs

Use flat representations while preserving hierarchy

Chose flat dictionaries or lists with hierarchy encoded in keys over deeply nested document trees or unstructured flattened text.

Rationale: Flat structures support batch ML operations, but keys retain enough document structure to recover context.

Invest most effort in retrieval

Chose retrieval quality and context selection over treating generation/model choice as the main lever.

Rationale: If retrieval is wrong, generation is based on wrong or stale information regardless of how fluent the generator is.

Tune chunk size instead of using a fixed universal chunk

Chose application-specific chunk sizes, overlap, and parent context over sentence-level or full-document embeddings.

Rationale: The right chunk must be narrow enough to search and broad enough to retain meaning.

Use summaries to improve retrieval or fit context

Chose summary embeddings or summaries of retrieved documents over passing full retrieved documents whenever possible.

Rationale: Summarization can reduce fluff, improve signal-to-noise, and fit more useful context into limited windows.

Use practical user constraints when they are cheaper than engineering perfection

Chose user guidance and reasonableness checks over six months of engineering to handle every bad prompt.

Rationale: Some failure modes can be reduced by telling users how to ask or by adding a simple pre-check.

Gotchas (what bit them)

The hardest part may be the box that looks easiest in the RAG diagram.

Root cause: Parsing and chunking must preserve document structure, exceptions, negations, tables, and heterogeneous file formats before retrieval can work.

Resolution: Spend time building parsers, heuristics, flat hierarchy-aware keys, and document-specific chunking logic before optimizing generation.

Naive chunking can invert or erase the meaning of policies and records.

Root cause: Rules, exclusions, and negations may live in parent sections or preceding words that are separated from the chunk.

Resolution: Represent hierarchy and include parent context or overlapping windows when chunks need surrounding scope.

Chunks that are too small or too large both damage retrieval.

Root cause: Sentence-level embeddings miss context, while document-level embeddings compress too much meaning into one vector.

Resolution: Experiment with chunk size, overlap, subsection-plus-parent context, and summary-based retrieval.

RAG can give a high-confidence stale answer with a source attached.

Root cause: The retrieval step may select an older or wrong document, and the generator will still produce confident text from it.

Resolution: Treat retrieval as the most important investment area and evaluate freshness, relevance, and evidence support.

Correctness metrics can hide retrieval failure.

Root cause: GPT-4-like models may know an answer from pretraining even when the retrieved context does not support it.

Resolution: Measure faithfulness separately from correctness using context-grounded evaluation.

Offline evaluation can miss real user behavior.

Root cause: Evaluation prompts may be easy and well-phrased, while users ask with different vocabulary and harder phrasing.

Resolution: Test query difficulty and user vocabulary, and consider lightweight product guidance or reasonableness checks.

Evaluation

The speakers evaluate RAG systems along multiple axes: whether retrieval selects the right context, whether the final answer is factually correct, whether the answer is faithful to the supplied context, whether prompts are easy or hard, and whether user behavior in production matches evaluation prompts. They also describe LLM-as-judge style faithfulness evaluation over question, answer, and context triples.

Metrics: Answer correctness · Faithfulness to retrieved context · Hallucination rate or inverse faithfulness · Retrieval relevance and freshness · Prompt/query difficulty · Sensitivity to user vocabulary · Multiple-choice benchmark accuracy when applicable

Outcomes

• The discussion identifies document parsing and chunking as the first major production RAG challenge.
• The team argues that retrieval quality is the most important part of a RAG application.
• The panel distinguishes answer correctness from faithfulness to retrieved context.
• The team lists concrete retrieval improvements: query rewriting, summaries, hybrid metadata/rules, fine-tuned embeddings, and adapters.
• The discussion reframes production RAG as data engineering, retrieval design, product guidance, and evaluation rather than a simple tutorial pipeline.

This is a practitioner discussion rather than a reported deployment metric. It includes concrete operating context such as thousands of pages, thousand-document context limits, a couple of days of PDF parser setup, five or six query rewrites, and the need to support real enterprise formats such as PDFs, Word documents, SharePoint sites, PowerPoint slides, wikis, emails, and tables.