System Prompt Learning for Coding Agents Using LLM-as-Judge Evaluation

arize · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Coding-agent optimization and evaluation

Arize needed a practical way to improve coding agents such as Claude and Cline through system prompts without the cost, sample hunger, and operational complexity of reinforcement learning or fine-tuning.

Constraints:

• Coding-agent behavior depends heavily on carefully crafted system prompts.
• Traditional reinforcement learning is sample-inefficient and resource-intensive.
• Prompt rules must be improved from concrete failure evidence rather than vibes.
• The approach had to work with a small benchmark set.
• Evaluation prompts themselves needed to produce useful explanations.

Architecture

Arize built an iterative system-prompt-learning loop: a coding agent solves benchmark issues with its current rules, execution results are evaluated by an LLM-as-judge prompt, the judge emits English-language failure feedback, and a meta-prompt turns that feedback into improved system prompt rules for the next iteration.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  system_prompt_rules["System prompt rules"] --> coding_agent["Coding agent"]
  coding_agent --> execution_harness["Execution harness"]
  execution_harness --> llm_as_judge_evaluator["LLM-as-judge evaluator"]
  llm_as_judge_evaluator --> meta_prompt_rule_generator["Meta-prompt rule generator"]
  meta_prompt_rule_generator --> system_prompt_rules

Components

Data flow

Claude or Cline receives a software engineering problem and current rules. The generated solution is executed against tests, the problem and execution results go to an LLM-as-judge evaluator, and the evaluator's explanatory feedback is fed into a meta-prompt that proposes updated prompt rules.

Why this architecture

The loop uses language feedback because prompt changes are cheap while RL and fine-tuning require much more data, infrastructure, and iteration cost.

Models used: Claude · Cline · LLM-as-judge · Meta-prompted LLM

Scale

• The approach was tested with 150 training examples.
  “Testing on the SWEBench benchmark with just 150 examples, they achieved a 5% improvement in GitHub issue resolution for Claude and 15% for Cline”summary, stated
• Reported gains were 5% for Claude and 15% for Cline on GitHub issue resolution.
  “they achieved a 5% improvement in GitHub issue resolution for Claude and 15% for Cline”summary, stated

Alternatives considered (roads not taken)

Reinforcement learning

Rejected because: It was framed as time intensive, data hungry, and requiring more specialized team support.

Model fine-tuning

Rejected because: System prompt learning avoided fine-tuning infrastructure while still improving coding-agent performance.

DSPy's MIPRO optimizer

Rejected because: The summary notes the comparison but also flags missing quantitative cost and iteration details.

Patterns

eval-driven-development (quality)

LLM-as-judge explanations drove each system prompt update for coding agents.

context-engineering (prompt and instruction management)

The system improved agent behavior by iteratively editing the instruction context supplied through Claude and Cline rule files.

Tradeoffs

Optimize system prompts instead of model weights.

Chose System prompt learning over RL or fine-tuning.

Rationale: It worked with small datasets and avoided heavier model-training infrastructure.

Generate English-language rules from judge feedback.

Chose Meta-prompted rule generation over Manual-only prompt editing.

Rationale: The loop could convert repeated failure modes into reusable prompt rules.

Gotchas (what bit them)

Scalar pass/fail rewards are too opaque for prompt improvement.

Root cause: The agent receives little information about why an attempt failed.

Resolution: Use an LLM-as-judge evaluator that emits detailed failure explanations.

Evaluation prompt quality becomes a dependency of the optimization loop.

Root cause: The learning signal is only as useful as the judge prompt's failure explanations.

Resolution: Invest in eval prompt engineering as a core competency.

Evaluation

Arize evaluated the prompt-learning loop on SWEBench with Claude and Cline, comparing issue-resolution improvements after applying generated rules.

Metrics: 150 SWEBench training examples. · 5% Claude improvement. · 15% Cline improvement. · Failure categories from LLM-as-judge explanations.

Outcomes

• Claude improved 5% on GitHub issue resolution.
• Cline improved 15% on GitHub issue resolution.
• The case demonstrated small-data prompt optimization without fine-tuning.

The benchmark used 150 training examples; the summary explicitly notes uncertainty about scaling to larger datasets and more diverse failure modes.