Infrastructure Noise in Agentic Coding Evaluations

anthropic · Research & Academia · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Reproducible evaluation infrastructure for agentic coding benchmarks

Anthropic found that agentic coding benchmark scores can shift materially based on Kubernetes resource enforcement, even when the model, harness, and task set stay fixed. Because agents install dependencies, run tests, and iterate inside live environments, CPU/RAM limits are part of the evaluation itself rather than a neutral container detail.

Constraints:

• leaderboard gaps between top models are often only a few percentage points
• agentic coding evals give models full runtime environments, not static output scoring
• Kubernetes guaranteed allocation and hard-limit settings can kill containers on transient spikes
• too little headroom causes spurious infrastructure failures
• too much headroom makes some solution strategies viable that fail under tighter limits
• time of day, cluster health, hardware specs, concurrency, and egress bandwidth can also confound scores
• benchmark consumers need interpretable model capability measurements

Architecture

Anthropic ran Terminal-Bench 2.0 on Google Kubernetes Engine while systematically varying resource configurations from strict 1x per-task specs to uncapped resources. All other variables stayed constant: same Claude model, same evaluation harness, and same task set. The experiments separated two effects: up to around 3x resource headroom, extra resources mostly reduced spurious infrastructure errors; beyond that, extra resources actively helped agents solve tasks by allowing heavyweight dependencies, subprocesses, and memory-intensive test suites. Anthropic recommends specifying both guaranteed allocations and hard kill thresholds, empirically calibrating the band so floor/ceiling scores are within noise, and treating resource configuration as a first-class experimental variable.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Terminal_Bench_2_0_tasks["Terminal-Bench 2.0 tasks"] --> Evaluation_harness["Evaluation harness"]
  SWE_bench_crossover_tasks["SWE-bench crossover tasks"] --> Evaluation_harness
  Claude_model["Claude model"] --> Evaluation_harness
  Evaluation_harness --> Kubernetes_containers["Kubernetes containers"]
  Google_Kubernetes_Engine_cluster["Google Kubernetes Engine cluster"] --> Kubernetes_containers
  Resource_configuration_matrix["Resource configuration matrix"] --> Guaranteed_allocation["Guaranteed allocation"]
  Resource_configuration_matrix --> Hard_kill_threshold["Hard kill threshold"]
  Guaranteed_allocation --> Kubernetes_containers
  Hard_kill_threshold --> Kubernetes_containers
  Kubernetes_containers --> Agent_solution_process["Agent solution process"]
  Agent_solution_process --> Result_analyzer["Result analyzer"]
  Hard_kill_threshold --> Infrastructure_error_tracker["Infrastructure error tracker"]
  Infrastructure_error_tracker --> Result_analyzer
  Evaluation_harness --> Result_analyzer
  Result_analyzer --> Benchmark_reporting_guidance["Benchmark reporting guidance"]

Components

Data flow

Terminal-Bench tasks run inside Kubernetes containers with per-task CPU/RAM specifications. The evaluation harness launches the same Claude model against the same task set under six resource configurations. Kubernetes reserves guaranteed resources and kills containers at the hard limit. Anthropic measures success rate and infrastructure error rate across configurations, compares statistical significance, replicates the direction across Claude variants, and tests SWE-bench with RAM up to 5x baseline. The resulting guidance is to report resource bands and enforcement methodology alongside benchmark scores.

Why this architecture

The controlled matrix isolates infrastructure as an experimental variable. By holding model, harness, and tasks fixed while varying resource floors/ceilings, Anthropic can distinguish spurious infrastructure failure reduction from genuine task-easiness changes caused by generous resources. The resulting architecture treats benchmark execution as an end-to-end system test, not a pure model-output measurement.

Models used: Claude model held constant in Terminal-Bench experiments · different Anthropic model variants · models other than Claude mentioned but not rigorously tested

Scale

• The gap between most- and least-resourced Terminal-Bench setups was 6 percentage points.
  “the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).”article, stated
• Anthropic observed infrastructure error rates as high as 6% during calibration.
  “infra error rates were surprisingly high: as many as 6% of tasks were failing because of pod errors”article, stated
• Strict enforcement produced a 5.8% infrastructure error rate.
  “infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped.”article, stated
• 3x headroom reduced infra errors from 5.8% to 2.1% with p < 0.001.
  “strict enforcement to 3x headroom (5.8% to 2.1%) was significant at p < 0.001.”article, stated
• From 1x through 3x, success scores were within noise with p=0.40.
  “From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40).”article, stated
• Between 3x and uncapped, infrastructure errors dropped 1.6 points while success jumped almost 4 points.
  “Between 3x to uncapped, infra errors drop an additional 1.6 percentage points, while success jumps almost 4 percentage points.”article, stated
• SWE-bench crossover varied RAM up to 5x baseline across 227 problems with 10 samples each.
  “We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each.”article, stated
• SWE-bench scores were 1.54 percentage points higher at 5x RAM than 1x.
  “Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x.”article, stated
• Anthropic recommends skepticism for leaderboard differences below 3 percentage points unless configuration is matched.
  “leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched.”article, stated

Alternatives considered (roads not taken)

Treat benchmark resource specs as both floor and hard ceiling

Rejected because: This leaves zero headroom for transient memory spikes and causes spurious OOM failures.

Run uncapped or highly generous resources

Rejected because: Generous allocations actively change what tasks agents can solve and can reward heavyweight strategies.

Collapse all resource configurations into one benchmark score

Rejected because: Different limits test different capabilities and make generalizability hard to interpret.

Trust small leaderboard gaps as pure model capability

Rejected because: A 2-point lead can reflect hardware, timing, or infrastructure differences rather than capability.

Patterns

eval-driven-development (quality)

Anthropic treats benchmark infrastructure as an experimental variable and calibrates resource bands empirically before interpreting scores.

observability-tracing (operations)

The investigation tracks infrastructure error rates separately from task success so pod/OOM failures are not mistaken for model failures.

safety-policy-enforcement (governance)

Resource limits are separated into guaranteed allocations and hard ceilings to avoid spurious failures while preventing score inflation.

cost-aware-routing (operations)

The study shows benchmark scores depend on resource budgets, so cost/resource allocation must be reported and matched when comparing models.

Tradeoffs

Resource ceiling calibration

Chose 3x-style headroom band for Terminal-Bench 2.0 over strict 1x enforcement.

Rationale: 3x reduced infra errors by about two-thirds while keeping score lift within noise.

Separate floor and ceiling

Chose guaranteed allocation plus hard kill threshold over single exact resource specification.

Rationale: The split gives enough breathing room to avoid OOM noise while preserving a hard ceiling against score inflation.

Benchmark interpretation

Chose treat small leaderboard gaps skeptically over interpret every 1-2 point delta as model capability.

Rationale: Moderate infrastructure spread and binomial uncertainty can already cover the gap.

End-to-end versus pure model measurement

Chose document infrastructure when using end-to-end agent evals over pretend runtime environment is irrelevant.

Rationale: Agentic evals necessarily test the whole runtime process, so infrastructure choices must be part of the measurement specification.

Gotchas (what bit them)

Strict resource enforcement causes false model failures.

Root cause: Kubernetes specs set as both guaranteed allocation and hard ceiling leave no headroom for transient memory spikes.

Resolution: Specify a guaranteed floor and separate hard kill threshold.

Too much headroom changes what the benchmark measures.

Root cause: Generous allocations let agents use heavyweight dependencies, subprocesses, and memory-intensive tests that fail under tight limits.

Resolution: Calibrate the resource band so the ceiling improves stability without making the task materially easier.

Different models' default strategies interact with infrastructure limits.

Root cause: Some agents install large libraries while others implement lean standard-library solutions.

Resolution: Report resource configuration so model rankings can be interpreted as system-dependent.

Leaderboard differences can be smaller than infrastructure noise.

Root cause: Resource allocation, cluster health, time limits, hardware, concurrency, egress, and time of day all affect end-to-end agent evals.

Resolution: Skepticism for sub-3-point gaps unless configurations are documented and matched.

Provider and external evaluator results may be asymmetric.

Root cause: Model providers can dedicate hardware and shield evals from variance, while outside evaluators may not be able to.

Resolution: Publish setup configurations and, where possible, run across multiple times/days.

Evaluation

Anthropic held model, harness, and task set constant while varying Terminal-Bench 2.0 resource configurations from 1x strict enforcement to uncapped. It tracked success rates, infrastructure error rates, statistical significance, and replicated directionality across Claude variants. It also ran a SWE-bench RAM crossover on 227 problems with 10 samples each.

Metrics: Terminal-Bench setup gap: 6 percentage points, p < 0.01 · infra errors: 5.8% strict to 0.5% uncapped · 1x to 3x infra errors: 5.8% to 2.1%, p < 0.001 · 1x through 3x success within noise, p=0.40 · 3x to uncapped: infra errors down 1.6 points, success up almost 4 points · SWE-bench: 227 problems, 10 samples each · SWE-bench 5x RAM versus 1x: +1.54 percentage points · leaderboard skepticism threshold: below 3 percentage points

Outcomes

• Resource configuration alone produced a 6-point Terminal-Bench spread.
• Moderate 3x headroom reduced spurious infrastructure failures while keeping score lift within noise.
• Uncapped resources began changing what the eval measured by enabling heavier solution strategies.
• SWE-bench showed the same directional RAM sensitivity, though smaller at 1.54 points.
• Anthropic recommends reporting guaranteed allocations, kill thresholds, resource bands, and enforcement methodology.
• Leaderboard gaps below 3 points should be treated skeptically unless evaluation infrastructure is documented and matched.

This is a rigorous evaluation-infrastructure study, not an application deployment. Its lesson transfers directly to production agent systems: runtime resources are part of the behavior envelope, and model capability cannot be separated cleanly from infrastructure without controlled measurement.