Evolution from Static Benchmarks to Adaptive Agent Evaluation Systems

comet · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: adaptive evaluation for agentic AI systems

Comet argues that static benchmarks and handcrafted eval sets calcify while agentic AI systems, user behavior, and production intent keep changing.

Constraints:

• AI applications are not static, but many evaluations treat them like static software.
• Agentic systems self-optimize around intent, making behavior harder to evaluate with fixed tests.
• Production traces, telemetry, cost, and failure data are needed for continuously updated evals.
• A persistent 20% of user behavior changes and can break businesses if the eval suite does not adapt.
• Organizations still need observability and online evaluation rather than believing evals are dead.

Architecture

Comet proposes malleable evals: evaluation systems that self-curate from production traces, optimize toward intent-based outcomes, run online and always-on checks, and feed telemetry back into agents or harnesses. Instead of a static data set as the eval, the desired end state and production trace stream become the source for adaptive tests and self-healing corrections.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-agent-traces["Production agent traces"] --> adaptive-eval-layer["Adaptive eval layer"]
  telemetry-stream["Telemetry stream"] --> adaptive-eval-layer
  static-benchmark-baseline["Static benchmark baseline"] --> adaptive-eval-layer
  adaptive-eval-layer --> agent-or-harness-correction-loop["Agent or harness correction loop"]
  agent-or-harness-correction-loop --> production-agent-traces

Components

Data flow

Production agents emit traces and telemetry about user intent, outcomes, cost, failures, and changed behavior. An adaptive eval layer analyzes those traces, self-curates or updates test suites, runs online evals, and feeds corrections or alerts back into the agent/harness and owners.

Why this architecture

The architecture treats evals as living software because agentic products are malleable. Static tests can still cover the stable 80%, but the changing 20% must be monitored through traces and telemetry so evals adapt as users and systems evolve.

Models used: Comet evals · Agentic AI systems · Production traces · Telemetry · Adaptive testing · Intent-based outcomes

Scale

• Comet benchmarks and runs evals for companies such as Uber, Netflix, and UK banks.
  “We benchmark and run evals for large set of companies and organizations, everything from like Uber to Netflix to to banks even in the UK.”transcript, stated
• The speaker frames about 80% of behavior as static and 20% as changing.
  “you can imagine this space where you go like 80% is like the static stuff”transcript, stated

Alternatives considered (roads not taken)

Static benchmarks and offline handcrafted evals only.

Rejected because: They fail to keep up with changing agent behavior, user intent, and shipped software.

Assume evals and observability are obsolete.

Rejected because: Agentic systems make measurement more important, not less.

Patterns

eval-driven-development (adaptive evals)

Comet reframes evals as continuously updated systems driven by intent, traces, and telemetry.

observability-tracing (trace-fed evals)

Production traces feed suite curation, adaptive tests, and online evaluation.

data-quality-loop (eval set maintenance)

Eval suites are updated from production behavior rather than remaining fixed after manual creation.

Tradeoffs

Keep static tests for stable behavior but add adaptive evals.

Chose Hybrid static baseline plus trace-fed adaptive suite over Only handcrafted offline evals.

Rationale: Stable behavior can remain defined, but changing behavior requires living evaluation.

Use telemetry in the loop.

Chose Cost, breakage, and harness-aware telemetry feeding evals over Eval datasets isolated from production observability.

Rationale: Harnesses can only self-heal or adapt if they know what is breaking and how expensive behavior is.

Gotchas (what bit them)

Static eval suites calcify.

Root cause: Applications and user behavior change faster than fixed benchmarks.

Resolution: Treat evals as malleable systems that update from traces and intent outcomes.

The changing 20% can break the business.

Root cause: Even if most behavior is stable, unpredictable user interactions keep shifting.

Resolution: Use agents and adaptive evals to monitor, maintain, and update coverage of that changing slice.

Intentful machines complicate measurement.

Root cause: Agents optimize toward user intent and personalized outcomes rather than fixed prompt-response pairs.

Resolution: Evaluate against intended end states and production outcomes.

Evaluation

Comet proposes moving from static benchmarks to intent-based outcomes, self-curated suites from traces, online always-on evaluations, and telemetry-in-the-loop corrections for agentic systems.

Metrics: Intent outcome alignment · Trace-derived failure coverage · Online evaluation results · Telemetry/cost signals · Static vs changing behavior coverage · Self-healing events

Outcomes

• The talk defines the eval-calcification problem and argues for adaptive evaluation systems.
• It proposes self-curated suites from traces, online eval optimization, and telemetry-in-the-loop agent correction.
• It reframes evals as living software or agents rather than static datasets.

The talk cites Comet's work across organizations including Uber, Netflix, and UK banks and frames future failures around the changing 20% of behavior.