Troubleshooting and Optimizing RAG Pipelines: Lessons from Production

lemonade · Insurance · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production RAG troubleshooting

Lemonade needed reliable RAG pipelines for customer-facing insurance automation, where failures can come from missing content, retrieval misses, re-ranker mistakes, or output-formatting problems.

Constraints:

• Lemonade serves millions of customers globally.
• Half of claims are handled fully automatically.
• RAG failures can cause hallucinations or non-informative answers.
• The pipeline has offline indexing and online query stages with different failure modes.
• Complex documents such as PDFs and tables can break standard parsing.

Architecture

The production RAG pipeline separates offline indexing from online querying. Documents are chunked, sometimes summarized by an LLM, embedded, and stored in a vector database. Queries are rewritten or embedded, top-k documents are retrieved, a re-ranker narrows them to top-n, and a generator/composer LLM writes the answer. Troubleshooting uses data cleaning, document-processing tools, critic checks, hyperparameter tuning, embedding fine-tuning, and query rewriting.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  knowledge_base["Knowledge base"] --> data_cleaning_and_document_processing["Data cleaning and document processing"]
  data_cleaning_and_document_processing --> chunker["Chunker"]
  chunker --> embedding_model["Embedding model"]
  embedding_model --> vector_database["Vector database"]
  query_rewriting_layer["Query rewriting layer"] --> retriever["Retriever"]
  vector_database --> retriever
  retriever --> re_ranker["Re-ranker"]
  re_ranker --> generator_llm["Generator LLM"]
  generator_llm --> critic_llm["Critic LLM"]

Components

Data flow

Offline, the knowledge base is cleaned, chunked, optionally summarized, embedded, and stored in a vector database. Online, the user question is rewritten or embedded, the retriever fetches top-k documents, the re-ranker selects top-n, and the generator composes the response, with critic checks used to detect ungrounded answers.

Why this architecture

The two-stage architecture makes RAG failures diagnosable: missing content is an indexing problem, retriever/re-ranker misses are retrieval problems, and hallucinated answers require prompt, critic, or context fixes.

Models used: embedding model · generator/composer LLM · critic LLM · re-ranker

Scale

• Lemonade serves millions of customers worldwide.
  “a technology-oriented insurance company that sells five insurance products to millions of customers worldwide.”summary, stated
• Lemonade has used chat as the primary customer interaction interface since 2015.
  “Since 2015, well before the current wave of generative AI, the company has used chat interfaces as the primary means of customer interaction”summary, stated
• 50% of claims are handled fully automatically.
  “Today, 50% of insurance claims submitted to Lemonade are handled fully automatically”summary, stated
• The example claim approval and reimbursement happened within two seconds.
  “claim approval and reimbursement within just two seconds.”summary, stated

Alternatives considered (roads not taken)

Treat every RAG failure as a generator problem

Rejected because: Failures can originate in missing content, retrieval, re-ranking, or formatting.

Return answers without a grounding check

Rejected because: A critic can prevent nonsense or ungrounded answers from reaching customers.

Use standard parsers for complex documents

Rejected because: PDFs and tables often require better document processing tools.

Patterns

rag-retrieval (retrieval pipeline)

The system uses a classic offline-index, online-query RAG architecture.

retrieval-ranking (candidate selection)

A re-ranker selects the top-n most relevant documents from retriever top-k results.

query-rewriting (retrieval improvement)

The query stage includes query rewriting or embedding before retrieval.

llm-as-judge (grounding check)

A critic LLM checks generated answers against question and context.

Tradeoffs

Add a critic call.

Chose Extra grounding confidence check over Returning every generated answer directly.

Rationale: The critic prevents ungrounded or non-informative responses from reaching customers.

Tune retriever parameters.

Chose Grid search over chunk size and k over Default chunking and retrieval settings.

Rationale: Retriever misses are sensitive to chunking and top-k choices.

Gotchas (what bit them)

Missing content produces hallucinations or no-answer behavior.

Root cause: The relevant content may not exist in the knowledge base or vector database.

Resolution: Clean data, improve document processing, and add explicit don't-know behavior.

Retriever and re-ranker failures require separate tests.

Root cause: The right document can be absent from top-k or lost during top-n re-ranking.

Resolution: Build an evaluation dataset with known query-document matches and test retriever and re-ranker independently.

Noisy scraped data harms downstream RAG.

Root cause: Real-world data contains contradictions, XML tags, and irrelevant content.

Resolution: Use data cleaning tools and better document parsers.

Evaluation

Lemonade recommends evaluating RAG with known query-document pairs, retriever top-k checks, re-ranker top-n checks, and critic confidence on generated answers.

Metrics: 50% of claims fully automated. · Two-second claim approval/reimbursement example. · Retriever top-k and re-ranker top-n recall on known query-document pairs.

Outcomes

• Lemonade documented a production troubleshooting map for RAG pipelines.
• The approach separates indexing, retrieval, re-ranking, generation, and critique failure modes.
• The company applies these lessons in a long-running automated insurance context.

Lemonade serves millions of customers and handles 50% of claims fully automatically.