Exploring RAG Limitations with Movie Scripts: The Copernicus Challenge

opengpa · Research & Academia · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: RAG evaluation for context-rich documents

OpenGPA appears to examine how standard RAG can fail on context-rich documents, especially when relevant information depends on broader narrative, temporal, or logical relationships rather than a directly retrievable keyword match.

Constraints:

• The original source was unavailable due to a DNS resolution error, so this extraction is summary-only and necessarily speculative.
• Context-rich documents can spread relevant information across non-adjacent sections.
• Standard chunking and embedding retrieval can lose broader narrative structure.
• A useful evaluation needs edge cases that test answer completeness, not only semantic chunk similarity.
• Production RAG systems need ways to detect and handle likely failure modes.

Architecture

Because the source was unavailable, the summary only supports a conceptual RAG-evaluation architecture: construct context-rich document tasks such as the Copernicus/movie-script challenge, test baseline chunk-and-embed retrieval, inspect failures where information is implicit or distributed, and use those findings to choose hierarchical retrieval, multi-stage retrieval, query reformulation, confidence scoring, or human evaluation.

Components

Why this architecture

The summary argues that ordinary RAG can work on simple queries while failing on context-dependent questions, so the appropriate system-design response is evaluation-first: expose the failure mode, then decide whether retrieval should become hierarchical, iterative, summary-aided, confidence-scored, or agentic.

Models used: Retrieval-Augmented Generation · Vector embeddings · Chunking · LLM answer generation · Query expansion · Hierarchical retrieval · Multi-stage retrieval

Alternatives considered (roads not taken)

Treat standard chunk-and-embedding RAG as sufficient.

Rejected because: The summary identifies failures where meaning depends on broader context, temporal relationships, or information spread across sections.

Use single-query retrieval for all questions.

Rejected because: Some questions may require query expansion, decomposition, or iterative retrieval across multiple document sections.

Measure only retrieval accuracy.

Rejected because: The summary says context-rich RAG evaluation should also capture answer completeness and subjective quality.

Patterns

rag-retrieval (baseline retrieval architecture)

The case centers on standard RAG over context-rich documents and its limits when chunk retrieval misses broader document relationships.

eval-driven-development (RAG edge-case testing)

The summary frames the work as identifying and characterizing RAG failure modes before production deployment.

query-rewriting (retrieval mitigation)

The summary proposes query expansion, decomposition, or iterative retrieval when a single query cannot capture all needed context.

confidence-thresholding (failure detection)

The summary identifies confidence scoring as a monitoring mechanism for likely RAG failures.

Tradeoffs

How to choose chunk size in context-rich documents.

Chose Treat chunk size as an evaluated tradeoff over Assuming one chunk size works across documents.

Rationale: Smaller chunks can retrieve precise facts but lose narrative context, while larger chunks can preserve context but reduce precision.

How to handle questions needing multiple sections.

Chose Query expansion, decomposition, or iterative retrieval over Single-query retrieval.

Rationale: A single retrieval pass may not collect all evidence needed for synthesis across a document.

How to make RAG reliable enough for production.

Chose Monitor likely failure modes and collect user feedback over Trust retrieval outputs without uncertainty handling.

Rationale: The summary argues production systems need query classification, confidence scoring, and feedback to detect where RAG may fail.

Gotchas (what bit them)

Chunking can remove the context needed to answer the question.

Root cause: Smaller chunks improve precision but lose surrounding structure; larger chunks preserve context but add irrelevant content.

Resolution: Evaluate chunk-size tradeoffs and consider hierarchical or multi-stage retrieval for context-rich documents.

Embedding similarity can miss relationships that span sections.

Root cause: Standard embeddings operate at sentence or paragraph level and may not represent temporal, logical, or argumentative structure.

Resolution: Use evaluation cases that require synthesis and consider retrieval strategies that preserve document structure.

The extraction itself cannot claim original findings from the unavailable source.

Root cause: The source URL returned a 404/DNS error and no specific experimental results were available.

Resolution: Mark the project summary-only and preserve uncertainty around methods, results, and recommendations.

Evaluation

The summary recommends evaluation with curated context-rich test sets, answer completeness metrics, human evaluation, and failure-mode analysis, but does not provide original benchmark results because the source was unavailable.

Metrics: Retrieval accuracy · Answer completeness · Human evaluation quality · Failure-pattern identification · Confidence score behavior

Outcomes

• The summary identifies context-rich RAG as an important limitation area rather than reporting concrete experimental results.
• It highlights architectural options such as hierarchical retrieval, multi-stage retrieval, summarization, and agentic iterative retrieval.
• It emphasizes monitoring, confidence scoring, and user feedback for production RAG failure detection.

No production scale is available because the original source content could not be accessed.