AI Agent Optimization: Using Claude to Systematically Improve Memory Extraction Quality

lerim · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Coding-agent memory extraction

Lerim needed to improve memory extraction quality without guessing which prompts, tools, schemas, or DSPy signatures mattered.

Constraints:

• golden eval dataset
• avoid overfitting
• cross-model optimization
• keep/revert discipline

Architecture

Lerim used Claude Code as an optimizer agent around another memory-agent system. The AutoResearch-style setup gives Claude Code program.md directives, modifiable prompts/tool descriptions/DSPy signatures/Pydantic schemas/harness code, and an immutable evaluator over golden cases. Each experiment modifies one surface, runs Lerim’s full or component pipeline, scores extraction/search/dedup/maintain quality, and keeps or reverts the change. Later rounds recalibrated the judge toward precision and quality alignment and separated the reusable optimization method from the specific objective.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  program-directives[Program directives] --> claude-code-optimizer[Claude Code optimizer]
  autoresearch-skill[AutoResearch skill] --> claude-code-optimizer
  claude-code-optimizer --> modifiable-optimization-surface[Modifiable optimization surface]
  modifiable-optimization-surface --> lerim-memory-pipeline[Lerim memory pipeline]
  lerim-memory-pipeline --> immutable-evaluator[Immutable evaluator]
  immutable-evaluator --> claude-code-optimizer
  immutable-evaluator --> lifecycle-validation[Lifecycle validation]

Components

Data flow

The author writes program.md constraints and objective, then lets Claude Code edit allowed Lerim surfaces. For each experiment, Claude changes a prompt, signature, schema, tool description, harness, or filter; the evaluator runs golden cases through the memory pipeline; composite metrics decide whether the change helped; Claude keeps successful changes and reverts failed ones. Component-level tests guide fast iteration, while lifecycle tests verify that improvements survive sequential sync and maintain cycles.

Why this architecture

The setup deliberately separates optimizer, editable surfaces, and immutable evaluator so another agent can make aggressive changes without moving the goalposts. Component-level evals keep the loop fast, lifecycle evals prevent local wins from failing in realistic memory accumulation, and keep/revert discipline turns failed experiments into knowledge instead of silent regressions.

Models used: Claude Code Opus 4.6 · MiniMax M2.5 · LLM judge

Scale

• Round 1 ran 14 experiments over 4.5 hours, keeping seven and discarding seven.
  “I ran 14 experiments over 4.5 hours. Seven were kept, seven were discarded.”article, stated
• The initial experiment produced a 41% composite quality improvement, with a one-line ChainOfThought change accounting for nearly half.
  “The result: 41% improvement in composite quality score from 14 experiments over one evening. And the single biggest win was changing one line of code.”article, stated
• Dedup accuracy improved from 0.28 to 0.72.
  “The dedup accuracy improvement is the most dramatic: from 0.28 to 0.72.”article, stated
• A later judge rebuilt around 327 golden cases used a deterministic layer and LLM judge layer.
  “The original composite score weighted extraction/search/dedup/maintain against 327 golden cases. The new metric is simpler and more honest: a deterministic layer (25%) checks structure, format, and content heuristics; an LLM judge layer (75%) scores `quality`”article, stated

Alternatives considered (roads not taken)

Prompt-only optimization

Rejected because: Schema/DSPy changes, especially ChainOfThought and field descriptions, beat prompt engineering alone.

Let the optimizer change the evaluator

Rejected because: The evaluator is immutable so Claude cannot improve scores by changing the target.

Use restrictive negative rules to raise precision

Rejected because: Restrictive rules repeatedly reduced recall without improving precision.

Patterns

eval-driven-development (quality)

Every agent change was measured against a golden dataset before being kept or reverted.

prompt-versioning (model-operations)

The optimization surface included prompts, tool descriptions, DSPy signatures, and schema field descriptions as explicit tunable artifacts.

Tradeoffs

How to use an agent to modify another agent.

Chose bounded experimental edits with evals and revert rules over unconstrained autonomous refactoring.

Rationale: The program file constrained what Claude could change and how success was measured.

Optimization safety boundary.

Chose modifiable surfaces plus immutable evaluator over unconstrained agent refactoring of system and metric.

Rationale: The optimizer can search broadly while the score remains stable and comparable.

"2. Modifiable code — Lerim’s prompts, tool descriptions, DSPy signatures, schema definitions, and the whole harness. Claude Code edits these.

3. Immutable evaluator— A composite score (extraction quality, search relevance, dedup accuracy, maintain precision)"

— article, stated

Guidance style for extraction quality.

Chose positive quality criteria and schema descriptions over restrictive skip lists and negative examples.

Rationale: Positive guidance improved quality while restrictive rules made the extractor cautious and reduced recall.

Gotchas (what bit them)

Prompt engineering was not the only or largest optimization lever.

Root cause: Schema and component-level changes could cascade through the end-to-end memory pipeline.

Resolution: Let the optimizer test prompts, signatures, schemas, tools, and harness changes under one metric.

Evaluation

Two rounds and 24 experiments measured composite quality score changes.

Metrics: 41% composite quality improvement · 24 experiments · 14 experiments in one evening

Outcomes

• The biggest individual gain came from switching DSPy Predict to ChainOfThought.

Author warns the exact code snapshot is old and not meant to be reproduced literally.