Security-Focused LLM Agent Harness for Automated Vulnerability Discovery

cloudflare · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-assisted vulnerability discovery

Cloudflare found that generic coding agents were poorly shaped for broad vulnerability research across large codebases, even when specialized security-focused models could reason about exploit chains and proofs.

Constraints:

• A single agent session covers only a tiny fraction of a large repository before context limits and compaction erase useful state.
• Security research requires many narrow hypotheses across many components, not one linear stream of work.
• Models over-report speculative bugs, which creates expensive triage noise.
• Confirmed findings must be proven exploitable and reachable from attacker-controlled input, not merely described.

Architecture

Cloudflare built a vulnerability-discovery harness around Mythos Preview and other frontier/security models. The harness first builds repository architecture context and a task queue, then runs many narrow hunter agents in parallel with scratch environments, sends findings to an independent adversarial reviewer, traces confirmed shared-library bugs across consumer repositories, generates new hunt tasks for reachable traces, and writes schema-validated reports into an ingest API.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Repository corpus"] --> B["Architecture scout agent"]
  B --> C["Task queue"]
  C --> D["Hunter agents"]
  D --> E["Exploration subagents"]
  D --> F["Scratch proof environment"]
  D --> G["Adversarial reviewer"]
  G --> H["Reachability tracer agents"]
  H --> C
  H --> I["Structured report writer"]
  I --> J["Vulnerability management process"]

Components

Data flow

An architecture agent reads a repository and emits trust-boundary context plus initial tasks; concurrent hunter agents investigate one attack class and scope at a time, using tools to compile and run proof-of-concept code in scratch directories; an independent reviewer tries to disprove each finding; tracer agents fan out across consumer repositories with a symbol index to test reachability; confirmed reachable findings become reports written to a predefined schema and submitted to Cloudflare's vulnerability process.

Why this architecture

Cloudflare built a harness because vulnerability research needs parallel, narrow, hypothesis-driven work, proof generation, adversarial filtering, and reachability tracing; a generic single-stream coding agent produced findings but not meaningful coverage or reliable value at scale.

Models used: Mythos Preview · frontier models · security-focused LLMs

Scale

• Cloudflare pointed Mythos Preview at more than 50 repositories.
  “We soon pointed it at more than fifty of our own repositories”article, stated
• Hunter agents typically ran around 50 at once.
  “Hunters (the agents that actually look for bugs) run concurrently, typically around fifty at once”article, stated
• A single generic agent covered maybe a tenth of a percent of a hundred-thousand-line repository usefully.
  “against a hundred-thousand-line repository can cover maybe a tenth of a percent of the surface in a useful way”article, stated
• The harness scanned Cloudflare runtime, edge data path, protocol stack, control plane, and open-source dependencies.
  “It was used to scan live code across our runtime, edge data path, protocol stack, control plane, and the open-source projects we depend on.”article, stated

Alternatives considered (roads not taken)

Point a generic coding agent at a repository

Rejected because: It can produce findings but cannot provide meaningful coverage of real codebases or valuable vulnerability triage at scale.

Let the initial finder check its own work

Rejected because: Cloudflare found adversarial review with a separate prompt/model reduced noise.

Ask one agent to judge bug existence and exploit reachability together

Rejected because: The model reasoned better when bug existence and attacker reachability were split into narrower questions.

Patterns

multi-agent-orchestration (agent-architecture)

Cloudflare decomposed vulnerability research into architecture, hunter, reviewer, tracer, and report-writing agents that each answer narrower questions.

tool-use-orchestration (agent-execution)

Hunter agents compile and run proof-of-concept code in per-task scratch directories instead of stopping at textual vulnerability descriptions.

eval-driven-development (quality-control)

The harness deliberately over-reports for recall, then uses adversarial review and schema validation to reduce noise before findings hit the queue.

Tradeoffs

Tune harnesses to over-report.

Chose higher recall with more noise over low-noise filtering that might miss bugs.

Rationale: Cloudflare preferred seeing more findings and reducing them later through review and triage.

Use many narrow agents in parallel.

Chose attack-class/scope-specific hunters over one exhaustive coding agent.

Rationale: Coverage improved when many agents worked on tightly scoped questions and results were deduplicated.

Separate adversarial review from finding generation.

Chose a reviewer agent with no ability to generate new findings over letting the finder self-review.

Rationale: Deliberate disagreement catches noise the initial finding agent would miss.

Gotchas (what bit them)

Generic agents produce shallow coverage on large repos.

Root cause: They hold one hypothesis at a time, fill context, and compact away useful state before covering enough attack surface.

Resolution: Use many narrow parallel hunters plus task queues and deduplication.

Model findings can flood triage with speculation.

Root cause: Models tend to find possible bugs whether or not solid evidence exists, and hedged findings compound human attention cost.

Resolution: Tune for recall, but add adversarial review and proof generation before queueing.

A bug description is not enough for security prioritization.

Root cause: Security impact depends on exploitability and whether attacker-controlled input reaches the flaw.

Resolution: Generate proof code and use tracer agents to test reachability across consumer repositories.

Model refusals were inconsistent for legitimate security work.

Root cause: The same task could produce different model behavior depending on framing, context, or run randomness.

Resolution: Treat refusals as an operational constraint of the harness rather than assuming a model will respond consistently.

Evaluation

Cloudflare evaluated models and the harness on its own infrastructure and repositories, emphasizing exploit-chain construction, proof generation, triage quality, hedged finding reduction, reachability, and formal vulnerability management outcomes.

Metrics: number of repositories scanned · parallel hunter count · hedged finding reduction · reproduction/proof quality · fix-or-dismiss triage effort · reachability from attacker-controlled input

Outcomes

• Mythos Preview could chain lower-severity primitives into more severe exploits and generate proofs.
• The harness scanned more than 50 repositories and live code across several Cloudflare systems.
• Adversarial review and reachability tracing improved triage quality before findings entered formal vulnerability management.

The harness typically ran around 50 hunters concurrently and treated vulnerability research as many narrow tasks across large repositories rather than one long agent session.