Benchmarking AI Agents for Software Bug Detection and Maintenance Tasks

bismuth · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Software-maintenance agent benchmarking

Bismuth needed a benchmark that tests whether coding agents can find, triage, and fix real maintenance bugs, not just solve upfront coding tasks on existing benchmarks.

Constraints:

• existing code benchmarks underrepresent maintenance, bug detection, and code review
• bugs need to be objective enough to compare across agents
• repository-wide search would be too slow and may bias benchmark design
• agents produce many false positives that human reviewers will not inspect
• bug finding requires cross-file reasoning, program comprehension, and targeted search

Architecture

Bismuth built SM-100 from 100 triaged, validated, classified bugs across more than 84 public repositories in Python, TypeScript/JavaScript, and Go. Each bug carries metadata for severity, domain knowledge, discovery difficulty, and implication. The benchmark measures discovery, false positive rate, introduction-time detection, and remediation. To avoid repository-wide search while avoiding hints, repositories are scoped to interrelated subsystems containing the golden changed files, then agents operate over that subsystem.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  bug-corpus[Bug corpus] --> metadata-annotations[Metadata annotations]
  bug-corpus --> subsystem-scoper[Subsystem scoper]
  subsystem-scoper --> agent-under-test[Agent under test]
  agent-under-test --> bug-report-grader[Bug report grader]
  agent-under-test --> remediation-evaluator[Remediation evaluator]
  metadata-annotations --> benchmark-metrics[Benchmark metrics]
  bug-report-grader --> benchmark-metrics
  remediation-evaluator --> benchmark-metrics

Components

Data flow

Curators gather remediated real-world bugs, annotate metadata, filter to objective crash/data-loss/security/logical issues, and build subsystem scopes around golden commits. Agents receive those scoped files, report suspected bugs, are manually assessed for false positives and true positives, and then attempt remediations for discovered bugs.

Why this architecture

The benchmark scopes tasks to subsystems so agents are not asked to search entire repositories, but it avoids directly hinting at the bug. It focuses on objective bugs so results are reproducible, and it separates discovery, false-positive rate, introduction-time review, and remediation because an agent that reports many bugs or fixes poorly is not actually useful for maintenance.

Models used: Bismuth · Claude Code · Codex · Devin · Cursor agent · Cosine · Lava Maverick

Scale

• SM-100 contains 100 bugs.
  “So, we painstakingly gathered 100 bugs that we triaged, validated, and classified in over 84 public repositories.”transcript, stated
• The bugs span over 84 public repositories.
  “classified in over 84 public repositories. These are bugs that have all been remediated already but are represented in the wild and work in open”transcript, stated
• Basic loop agents found five or six bugs but had a 97% false positive rate.
  “We actually found five or six in the basic loop here. Uh, but they also had a 97% false positive rate. So, almost every basic agent that just”transcript, stated
• Bismuth found 10 needles while the next leading solution found seven.
  “we found 10 of our needles. Uh, the next leading solution finding seven. So we do to grow here but if anything we're”transcript, stated
• Claude Code, Bismuth, and Codex true-positive rates were reported at 16%, 25%, and 45%.
  “and detection though uh with claude code starting out at 16% us at 25% in second place and codeex at 45%. You'll also”transcript, stated

Alternatives considered (roads not taken)

Rely on existing coding benchmarks

Rejected because: They do not adequately cover maintenance, bug finding, deployment, or code review tasks.

Ask agents to search entire repositories

Rejected because: Full-repository search would take too long for a single target bug.

Benchmark ambiguous style, optimization, or feature issues

Rejected because: Humans debate those and they reduce reproducibility.

Patterns

eval-driven-development (benchmarking)

SM-100 provides a maintenance benchmark that exposes bug-detection and remediation gaps for agents.

retrieval-ranking (code-understanding)

Subsystem scoping narrows relevant files so agents can inspect likely related code without direct hints.

human-in-the-loop-gate (grading)

False positives and bug validity require human-style/manual grading rather than raw report counts.

Tradeoffs

Scope repositories to subsystems

Chose interrelated file subsets around golden commits over full repository search.

Rationale: This keeps the benchmark tractable without revealing the bug.

Use objective bugs only

Chose security/logical/data-loss/crash defects over feature requests, optimization, or debatable issues.

Rationale: Objective bugs improve reproducibility across evaluations.

Measure false positives as well as discoveries

Chose true-positive and report-volume metrics over rewarding raw bug-report counts.

Rationale: A flood of reports is unusable for engineers even if a real bug is buried inside.

Gotchas (what bit them)

Basic tool-loop agents report overwhelming false positives.

Root cause: Equipping a model with shell/search/replace/report tools is not enough for bug triage.

Resolution: Bismuth treats false-positive rate as a first-class benchmark metric.

Agents are too narrow in bug reasoning.

Root cause: Even thinking models inspect files through limited chains and miss alternative bug interpretations.

Resolution: The talk argues for broader thinking chains, deeper selected-chain reasoning, and better context use.

Software maintenance is harder than creating new software upfront.

Root cause: Maintenance requires targeted search, cross-file reasoning, bug-pattern recognition, and deployed-system comprehension.

Resolution: SM-100 focuses on the post-deployment maintenance problem explicitly.

Evaluation

The benchmark measures bug discovery without prior knowledge, false positive rate, PR/commit introduction-time detection, and remediation quality, using a curated multilingual corpus and subsystem scoping.

Metrics: 100 bugs over 84+ repositories · four benchmark numbers: discovery, false positives, introduction-time detection, remediation · 97% false positive rate for basic loop agents · Bismuth 10 needles vs next leading 7 · Claude Code 16%, Bismuth 25%, Codex 45% true-positive rate

Outcomes

• SM-100 exposes a maintenance gap not captured by higher scores on software-engineering creation benchmarks.
• Basic agents have extremely high false positive rates on bug detection.
• Bismuth and Codex showed stronger results on some bug-detection dimensions, but the talk frames the benchmark as unsaturated.
• The benchmark highlights the need for targeted search, deeper program comprehension, and cross-file reasoning.

The source gives clear benchmark scale and results: 100 bugs, 84+ repositories, four metrics, 97% false positives for basic agents, and named true-positive/discovery rates.