Context Rot: Evaluating LLM Performance Degradation with Increasing Input Tokens

chromadb · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Long-context LLM evaluation

ChromaDB needed to test whether long-context LLMs actually process information uniformly as input length grows. Widely used Needle-in-a-Haystack benchmarks made long context look solved, but they mostly test direct lexical retrieval rather than realistic semantic ambiguity, distractors, conversational memory, or long output reproduction.

Constraints:

The study needed to isolate input length from task difficulty.
Benchmarks had to cover semantic retrieval, distractors, haystack structure, conversational memory, and output-length scaling.
The evaluation had to compare many current closed and open models.
Long context tasks can degrade in non-uniform and model-specific ways.
Judge labels needed human calibration before automated scoring could be trusted.

Architecture

ChromaDB built a controlled evaluation suite across 18 LLMs. It extends Needle-in-a-Haystack with semantic needle-question similarity, distractor variants, needle-haystack similarity, and haystack-structure changes; adds LongMemEval focused-vs-full prompt comparisons; adds repeated-word replication; and calibrates LLM judges against manually labeled model outputs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  needle-in-a-haystack-extensions["Needle-in-a-Haystack extensions"] --> model-evaluation-runner["Model evaluation runner"]
  embedding-similarity-scorer["Embedding similarity scorer"] --> needle-in-a-haystack-extensions
  longmemeval-comparator["LongMemEval comparator"] --> model-evaluation-runner
  repeated-words-task["Repeated-words task"] --> model-evaluation-runner
  model-evaluation-runner --> llm-judge-calibration["LLM judge calibration"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Needle-in-a-Haystack extensions`	Controlled semantic retrieval tests that vary needle-question similarity, distractors, haystack similarity, and structure.	`Model evaluation runner`, `Embedding similarity scorer`
`Embedding similarity scorer`	Computes needle-question and needle-haystack similarity across multiple embedding models.	`Needle-in-a-Haystack extensions`
`LongMemEval comparator`	Compares focused prompts with full long prompts for conversational memory tasks.	`Model evaluation runner`
`Repeated-words task`	Tests whether models can exactly reproduce long repeated sequences as input and output length grow.	`Model evaluation runner`
`Model evaluation runner`	Runs the tested models across lengths, conditions, and model-specific settings.	`Needle-in-a-Haystack extensions`, `LongMemEval comparator`, `Repeated-words task`, `LLM judge calibration`
`LLM judge calibration`	Compares GPT-4.1 judge labels against manually labeled outputs before using automated evaluation.	`Model evaluation runner`

Data flow

The evaluation creates controlled prompt variants where input length changes while task difficulty is held as constant as possible. Models run across multiple input lengths and positions; outputs are scored with calibrated judges or task-specific checks; and results are compared by condition to reveal degradation from length, distractors, structure, and output-growth effects.

“Each task remains intentionally simple and is deliberately controlled to isolate the impact of context length alone.”article, stated

“We evaluate each model across its maximum context window with temperature=0 unless that setting is incompatible”article, stated

Why this architecture

The suite attacks the main confound in long-context evaluation: longer prompts often also make tasks harder. By varying one factor at a time and including focused-vs-full comparisons, the evaluation makes context length and context composition visible as reliability variables.

“our experiments hold task complexity constant while varying only the input length—allowing us to directly measure the effect of input length alone.”article, stated

“This performance drop suggests that adding irrelevant context, and thereby adding an additional step of retrieval, significantly impacts a model’s ability to maintain reliable performance.”article, stated

Models used: GPT-4.1 · Claude Opus 4 · Claude Sonnet 4 · Gemini 2.5 Pro · Gemini 2.5 Flash · Qwen3 models · GPT-4.1 judge

Scale

The study evaluated 18 LLMs.
“An evaluation across 18 LLMs, including leading closed-source and open-weights models, revealing nonuniform performance with increasing input length.”article, stated
The NIAH extension used 8 input lengths for each unique combination.
“8 input lengths”article, stated
The NIAH extension used 11 needle positions for each unique combination.
“11 needle positions”article, stated
LongMemEval full prompts averaged about 113k tokens.
“filtering out 38 prompts to end up with 306 total prompts. These prompts average out to ~113k tokens.”article, stated
The repeated-words task included 1090 context-length and unique-word-index variations.
“For a given word combination, we create 1090 variations of context lengths and unique word indices:”article, stated
Judge calibration used about 500 manually labeled NIAH outputs and about 600 LongMemEval outputs.
“A subset of model outputs are manually labeled as incorrect/correct (~500 outputs for NIAH, ~600 outputs for LongMemEval)”article, stated

Alternatives considered (roads not taken)

Treat standard Needle-in-a-Haystack results as sufficient

Rejected because: Standard NIAH mainly measures direct lexical matching and can overstate long-context reliability.

“While scalable, this benchmark typically assesses direct lexical matching, which may not be representative of flexible, semantically oriented tasks.”article, stated

Use benchmarks where longer inputs also make tasks harder

Rejected because: That confounds input length with task complexity.

“Tasks often grow in complexity as input length increases, making it difficult to isolate whether performance drops are due to longer inputs or inherently harder problems.”article, stated

Evaluate conversational memory only with full histories

Rejected because: Focused prompts are needed as a control to separate reasoning ability from retrieval over irrelevant context.

“We verify that the models are highly capable of succeeding on the focused inputs, then observe consistent performance degradation with the full inputs.”article, stated

Patterns

`eval-driven-development` (evaluation)

ChromaDB built controlled long-context tests to expose reliability failures hidden by simpler benchmarks.

“We design four controlled experiments to investigate the influence of these factors:”article, stated

`llm-as-judge` (evaluation)

The evaluation uses GPT-4.1 as an automated judge after comparing it against manually labeled subsets.

“GPT-4.1 is used to label the same subset of model outputs as incorrect/correct.”article, stated

`context-compaction` (context)

The LongMemEval comparison treats focused prompts as a compacted alternative to full chat histories.

“Focused input, containing only the relevant parts and so the model just has to do simple reasoning.”article, stated

Tradeoffs

Hold task complexity constant

Chose controlled synthetic and semi-synthetic tasks over more realistic but confounded long-context tasks.

Rationale: The design isolates input length as the variable under study.

“To address this, our experiments hold task complexity constant while varying only the input length”article, stated

Use semantic NIAH variants

Chose needle-question similarity and distractor tests over standard lexical needle retrieval only.

Rationale: Real applications require inferring relevance, not just matching exact words.

“In real-world applications, models are often expected to handle ambiguous tasks and identify relevant information without relying on exact lexical matches.”article, stated

Include both focused and full LongMemEval prompts

Chose paired controls for reasoning-only versus retrieval-plus-reasoning over scoring full long prompts alone.

Rationale: The comparison exposes the cost of irrelevant context.

“Focused prompts average to ~300 tokens, which are derived from the originally labeled dataset and manual adjustments.”article, stated

Gotchas (what bit them)

Long-context benchmarks can look solved when they test only lexical retrieval.

Root cause: Standard NIAH asks models to find a known sentence rather than reason semantically through ambiguous content.

Resolution: Add semantic needle-question similarity and distractor conditions.

“Models typically perform well on NIAH, which has led to the perception that long-context is largely solved.”article, stated

Distractors degrade performance non-uniformly.

Root cause: Topically related but wrong context can be more misleading than irrelevant context, and effects differ across models.

Resolution: Test single and multiple distractor conditions across input lengths.

“Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further.”article, stated

Haystack structure affects long-context performance.

Root cause: Models are sensitive to the logical flow and structure of context, not just whether relevant information is present.

Resolution: Compare original coherent haystacks with shuffled versions.

“Across all 18 models and needle-haystack configurations, we observe a consistent pattern that models perform better on shuffled haystacks than on logically structured ones.”article, stated

Adding full irrelevant chat history hurts memory tasks.

Root cause: The model must retrieve relevant parts before reasoning, adding a second task inside the same call.

Resolution: Compare full prompts against focused prompts that contain only relevant parts.

“In this case, the model has to perform retrieval across the long context in addition to reasoning.”article, stated

Evaluation

The project is itself an evaluation suite spanning semantic NIAH, distractors, haystack structure, LongMemEval, repeated words, and calibrated LLM judging.

Metrics: Performance by input length · Performance by needle-question similarity · Distractor impact · Focused versus full prompt accuracy · Repeated-word position and word-count accuracy · Judge agreement against manual labels

“Across all experiments, model performance consistently degrades with increasing input length.”article, stated

Outcomes

LLM performance degraded with longer inputs even under simple controlled conditions.
Lower needle-question similarity increased degradation.
Distractors and haystack structure affected models non-uniformly.
Full LongMemEval prompts performed worse than focused prompts.
Repeated-word replication became less reliable as context length grew.

The suite evaluated 18 LLMs, 306 cleaned LongMemEval prompts averaging about 113k tokens, and 1090 repeated-word variations.

“Through our experiments, we demonstrate that LLMs do not maintain consistent performance across input lengths.”article, stated

“Even on tasks as simple as non-lexical retrieval or text replication, we see increasing non-uniformity in performance as input length grows.”article, stated