AI-Powered Automated Issue Resolution Achieving State-of-the-Art Performance on SWE-bench

trae · Tech · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Automated software issue resolution benchmark performance

Trae aimed to improve SWE-bench Verified issue resolution by combining code agents, multiple LLMs, regression testing, and patch selection rather than relying on a single model attempt.

Constraints:

• benchmark rather than production workflow
• multiple plausible patches per issue
• selector performance degradation at larger sample sizes
• need to filter failing regression tests
• vendor leaderboard framing

Architecture

Trae scales SWE-bench solving by generating diverse candidate patches with multiple LLM coder agents, filtering failures through regression tests, applying syntax-based voting, and using a selector agent with dual verification for final patch choice.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  issue-intake["Issue intake"]
  coder-agents["Coder agents"]
  multi-model-generator-pool["Multi-model generator pool"]
  regression-testing-module["Regression testing module"]
  candidate-patch-pool["Candidate patch pool"]
  syntax-based-voting["Syntax-based voting"]
  selector-agent["Selector agent"]
  final-patch-output["Final patch output"]
  issue-intake --> coder-agents
  multi-model-generator-pool --> coder-agents
  coder-agents --> candidate-patch-pool
  regression-testing-module --> candidate-patch-pool
  regression-testing-module --> selector-agent
  candidate-patch-pool --> selector-agent
  syntax-based-voting --> selector-agent
  selector-agent --> final-patch-output
  final-patch-output --> issue-intake

Components

Data flow

Issue tasks flow to multi-model coder agents with editing, shell, code-graph, and reasoning tools; regression tests filter failures; syntax voting and a selector agent choose the final patch.

Why this architecture

The architecture exists because one model attempt leaves performance on the table, while naive LLM selection degrades as candidate-patch sample size grows.

Models used: Claude 3.7 Sonnet · Gemini 2.5 Pro · OpenAI o4-mini · OpenAI o1 selector in baseline

Scale

• The approach reported 70.6% success on SWE-bench Verified and a #1 leaderboard result in the cited configuration.
  “Our approach raises the overall success rate on the SWE-bench-verified benchmark to 70.6%”article, stated

  “achieved #1 with a score of 70.6% on the SWE-bench Verified when evaluated with Claude 3.7 and next only to Tools when the leaderboard includes results from Claude 4.”article, stated
• Trae reports a 70.6% resolve rate on SWE-bench Verified.
  “Our approach raises the overall success rate on the SWE-bench-verified benchmark to 70.6%”article, stated
• Claude, Gemini, and o4-mini each covered different resolve-rate ranges, motivating candidate diversity.
  “The resolve rate of Claude-3.7-Sonnet ranges from 60.6% to 62.6% * The resolve rate of Gemini-2.5-Pro-0506 ranges from 52.4% to 55% * The resolve rate of OpenAI o4-mini ranges from 54.4% to 55.8%”article, stated
• The simple LLM selector peaked at a sampling space of five or six before declining.
  “Although the Optimal Rate steadily increases, the performance of LLM-as-a-Selector peaks when the sampling space is 5 or 6, then starts to decline.”article, stated

Alternatives considered (roads not taken)

Single model, single patch attempt

Rejected because: Trae uses multiple models and larger sampling to preserve test-time scaling benefits.

Trust syntax consensus alone

Rejected because: The selector uses dual verification and can discard syntax-voting results when uncertain.

Keep every regression-failing candidate patch in selection

Rejected because: Candidate patches that failed regression tests were removed before final selection.

LLM selector alone over a larger patch pool

Rejected because: Trae added syntax voting and dual verification because selector performance degraded as sampling increased.

Patterns

tool-use-orchestration (orchestration)

The coding agent is given file-editing, shell, code knowledge graph, and reasoning tools to inspect repositories and produce patches.

model-routing (model-ops)

Candidate patches are generated by multiple models to diversify the solution pool before selection.

eval-driven-development (evaluation)

Candidate patches that fail relevant regression tests are removed before the selector chooses among remaining patches.

llm-as-judge (evaluation)

A selector agent judges candidate patches using issue context, project code, tools, syntax consensus, and multi-selection voting.

Tradeoffs

How to improve benchmark success beyond a single model attempt.

Chose test-time scaling with multiple generated patches and a selector agent over one best-model patch attempt.

Rationale: The system increases the chance that at least one candidate is correct, then invests in selection quality.

How to diversify candidate patches.

Chose multiple LLMs as Coder agents over sampling only one model.

Rationale: Different model families produce diverse patch candidates that can be filtered and voted over.

Gotchas (what bit them)

A simple LLM selector did not scale cleanly with more candidate patches.

Root cause: As the sample space increased, the optimal candidate pool improved but selector performance peaked and then declined.

Resolution: Add a dedicated selector pipeline with syntax voting, deduplication, multi-selection voting, and verification.

Syntactic consensus alone is not enough to trust a generated patch.

Root cause: Multiple agents can converge on similar-looking code that still may not satisfy the issue.

Resolution: Use the Selector agent's dual verification against context and tools before accepting syntax-voted patches.

Evaluation

Trae evaluates on SWE-bench Verified and compares single-attempt model resolve rates, selector behavior by sample size, and final ensemble performance.

Metrics: 70.6% resolve rate on SWE-bench Verified with Claude 3.7 framing · Claude 3.7 Sonnet single-attempt range 60.6% to 62.6% · Gemini 2.5 Pro single-attempt range 52.4% to 55% · OpenAI o4-mini single-attempt range 54.4% to 55.8%

Outcomes

• Trae reports a 70.6% SWE-bench Verified result from a multi-stage patch generation, filtering, and selection system.
• The reusable system lesson is the orchestration of multiple models and selectors, not a production issue-resolution deployment.

Vendor benchmark post; source qualifies the ranking as with Claude 3.7 and next only to Tools when Claude 4 results are included.