Evaluating Context Compression Strategies for Long-Running AI Agent Sessions

factory_ai · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: context compression for long-running software agents

Factory AI needed to evaluate whether compressed context preserves the facts, files, decisions, and next steps that long-running coding agents need after hundreds of messages and millions of tokens.

Constraints:

Long-running agent sessions can exceed any model's working-memory window.
Aggressive compression can cause agents to forget files, decisions, error messages, and dead ends.
Traditional summary metrics such as ROUGE do not measure whether an agent can continue the task.
Artifact tracking remains difficult even with structured summaries.
Compression quality must be balanced against total tokens needed to finish the task, not only per-request token count.

Architecture

Factory built a probe-based evaluation framework for context compression. Real production software-engineering sessions are compressed with three methods: Factory anchored iterative summarization, OpenAI /responses/compact, and Anthropic Claude SDK compression. Four probe types ask compressed agents to recall facts, artifacts, continuation state, and decisions, and GPT-5.2 grades responses across six rubric dimensions without knowing which compression method produced the answer.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  production-session-corpus["Production session corpus"] --> compression-methods["Compression methods"]
  compression-methods --> probe-generator["Probe generator"]
  probe-generator --> probe-response-runner["Probe response runner"]
  probe-response-runner --> llm-judge["LLM judge"]
  llm-judge --> compression-scorecard["Compression scorecard"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Production session corpus`	Real long-running debugging, PR review, feature, CI, data science, and ML research sessions.	`Compression methods`
`Compression methods`	Factory, OpenAI, and Anthropic compression strategies applied to identical conversation prefixes.	`Probe generator`
`Probe generator`	Creates recall, artifact, continuation, and decision probes from truncated context.	`Probe response runner`
`Probe response runner`	Asks the compressed context to answer probes requiring preserved details.	`LLM judge`
`LLM judge`	Scores probe responses with rubrics for accuracy, completeness, context awareness, artifact trail, reasoning, and instruction following.	`Compression scorecard`
`Compression scorecard`	Compares quality and compression efficiency across methods.	—

Data flow

Production agent sessions are sampled at compression points, the same conversation prefix is compressed by Factory, OpenAI, and Anthropic methods, four probes are generated from the truncated history, each compressed context is used to answer the probes, and GPT-5.2 grades each response against explicit rubrics.

“We designed a probe-based evaluation that directly measures functional quality.”article, stated

“The judge receives the probe question, the model's response, the compacted conversation context, and (when available) ground truth.”article, stated

Why this architecture

Factory chose probe-based evaluation because lexical similarity does not tell whether compressed context lets an agent keep working. Anchored structured summaries are compared against opaque and regenerated summaries so the evaluation can separate compression ratio from preservation of files, decisions, and task state.

“Traditional metrics like ROUGE or embedding similarity do not tell you whether an agent can continue working effectively after compression.”article, stated

“The key insight is that structure forces preservation.”article, stated

Models used: Factory anchored iterative summarization · OpenAI /responses/compact · Anthropic Claude SDK compression · GPT-5.2 LLM judge

Scale

The evaluation used over 36,000 production-session messages.
“We evaluated all three methods on over 36,000 messages from production sessions spanning PR review, testing, bug fixes, feature implementation, and refactoring.”article, stated
The methodology details list 36,611 messages.
“Dataset: Hundreds of compression points over 36,611 messages.”article, stated
A concrete debugging example covered 178 messages and 89,000 tokens.
“Consider a debugging session where a user reported a 401 error on the /api/auth/login endpoint despite valid credentials. Over 178 messages and 89,000 tokens”article, stated
OpenAI compressed 99.3% of tokens, Anthropic 98.7%, and Factory 98.6%.
“OpenAI compresses to 99.3% (removing 99.3% of tokens), Anthropic to 98.7%, Factory to 98.6%.”article, stated

Alternatives considered (roads not taken)

Optimize for smallest possible summaries.

Rejected because: Aggressive compression can lose the details that prevent re-reading files and repeating dead ends.

“The naive solution is aggressive compression: squeeze everything into the smallest possible summary.”article, stated

Use lexical similarity metrics such as ROUGE.

Rejected because: A summary can overlap lexically while missing the exact file path or decision the agent needs.

“A summary might score high on lexical overlap while missing the one file path the agent needs to continue.”article, stated

Use opaque compression only.

Rejected because: OpenAI's compact endpoint compresses strongly but is not inspectable for what it preserved.

“It achieves the highest compression ratios (99.3%) but sacrifices interpretability. You cannot read the compressed output to verify what was preserved.”article, stated

Patterns

`context-compaction` (agent memory)

Factory compares compression methods for preserving long-running agent state across context-window limits.

“We compared three production-ready compression strategies.”article, stated

`eval-driven-development` (compression evaluation)

Probe-based evals measure whether compressed context supports continuation rather than just summary similarity.

“Finally, probe-based evaluation captures something that traditional metrics miss.”article, stated

`llm-as-judge` (rubric grading)

GPT-5.2 scores each probe response against detailed rubrics and is blinded to the compression method.

“The judge does not know which compression method produced the response. It evaluates purely on response quality against the rubric.”article, stated

`agent-memory` (persistent state)

Factory keeps a persistent structured summary with sections for intent, file modifications, decisions, and next steps.

“Factory maintains a structured, persistent summary with explicit sections for different information types: session intent, file modifications, decisions made, and next steps.”article, stated

Tradeoffs

Use structured summaries.

Chose Anchored iterative summarization with explicit sections over Freeform generic summarization.

Rationale: Explicit sections force the summarizer to preserve files, decisions, intent, and next steps.

“By dedicating sections to specific information types, the summary cannot silently drop file paths or skip over decisions.”article, stated

Accept slightly lower compression for better retained context.

Chose Factory's 98.6% compression with higher quality over OpenAI's 99.3% compression with lower quality score.

Rationale: The small token-retention cost produced higher continuation quality and fewer lost technical details.

“Factory retains about 0.7% more tokens than OpenAI, but gains 0.35 quality points.”article, stated

Gotchas (what bit them)

Compression ratio can be the wrong target.

Root cause: The highest compression can discard details that later force the agent to re-fetch context or repeat work.

Resolution: Optimize for total tokens to complete the task and quality of preserved context.

“Compression ratio turned out to be the wrong metric entirely. OpenAI achieves 99.3% compression but scores 0.35 points lower on quality.”article, stated

Artifact tracking remains weak.

Root cause: General summarization loses or blurs file-created, file-modified, and file-examined state.

Resolution: Factory suggests specialized artifact handling beyond summarization, such as a separate artifact index.

“Artifact tracking remains an unsolved problem. All methods scored between 2.19 and 2.45 out of 5.0 on knowing which files were created, modified, or examined.”article, stated

Regenerating summaries can cause detail drift.

Root cause: Repeated full-summary regeneration may drop earlier facts and decisions.

Resolution: Factory incrementally merges newly truncated spans into a persistent summary.

“Factory's advantage comes from the anchored iterative approach: by merging new summaries into a persistent state rather than regenerating from scratch”article, stated

Evaluation

Factory generates recall, artifact, continuation, and decision probes from truncated sessions, then uses GPT-5.2 to grade answers on six rubric dimensions with weighted criteria and method-blind scoring.

Metrics: Overall quality score · Accuracy · Context awareness · Artifact trail · Completeness · Instruction following · Compression ratio

“For each compression point, we generated four probes (recall, artifact, continuation, decision) based on the truncated conversation history.”article, stated

“Each dimension has 2-3 criteria with explicit scoring guides.”article, stated

Outcomes

Factory scored 3.70 overall, compared with Anthropic at 3.44 and OpenAI at 3.35.
Factory scored 4.04 on accuracy, compared with Anthropic at 3.74 and OpenAI at 3.43.
Factory retained more technical details such as file paths and error messages while preserving similar compression efficiency.
Artifact trail remained weak for all methods and was identified as needing specialized handling.

Factory evaluated hundreds of compression points over 36,611 production messages and reported compression ratios near 98.6-99.3%.

“Factory scores 0.35 points higher than OpenAI and 0.26 higher than Anthropic overall.”article, stated

“Factory scores 4.04, Anthropic 3.74, OpenAI 3.43. The 0.61 point difference between Factory and OpenAI reflects how often technical details like file paths and error messages survive compression.”article, stated