AI-Powered Vulnerability Triage Using GitHub Security Lab Taskflow Agent

github · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Security alert triage

GitHub Security Lab needed to triage large volumes of CodeQL code scanning alerts where false positives often depend on fuzzy code semantics that humans can spot but static rules struggle to encode.

Constraints:

• security triage has many repetitive steps but also requires semantic reasoning about code
• large prompts can skip steps or exceed context limits
• deterministic information gathering should not be delegated to a non-deterministic LLM
• human auditors still need enough detail to verify results
• the agent is not given runtime exploit environments or static/dynamic analyzers beyond CodeQL alert generation

Architecture

GitHub Security Lab built taskflows as YAML-defined sequences of LLM tasks. A taskflow fetches CodeQL alert results, runs templated tasks across alerts asynchronously, gathers scoped notes into a database, audits likely false-positive causes, and generates GitHub issues or reports with source references. MCP servers handle deterministic file fetching, searching, workflow trigger extraction, and other programmatic checks. The LLM, mostly Claude Sonnet 3.5 in the reported experiments, handles fuzzy semantic reasoning such as whether authentication or sanitization makes an alert non-exploitable.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  codeql-alert-source["codeql-alert-source"] --> taskflow-runner["taskflow-runner"]
  taskflow-yaml-definitions["taskflow-yaml-definitions"] --> taskflow-runner["taskflow-runner"]
  taskflow-runner["taskflow-runner"] --> mcp-tool-servers["mcp-tool-servers"]
  taskflow-runner["taskflow-runner"] --> llm-triage-reasoner["llm-triage-reasoner"]
  taskflow-runner["taskflow-runner"] --> audit-notes-database["audit-notes-database"]
  mcp-tool-servers["mcp-tool-servers"] --> source-code-repository["source-code-repository"]
  source-code-repository["source-code-repository"] --> llm-triage-reasoner["llm-triage-reasoner"]
  llm-triage-reasoner["llm-triage-reasoner"] --> audit-notes-database["audit-notes-database"]
  llm-triage-reasoner["llm-triage-reasoner"] --> report-generation-step["report-generation-step"]
  audit-notes-database["audit-notes-database"] --> report-generation-step["report-generation-step"]
  report-generation-step["report-generation-step"] --> human-security-auditor["human-security-auditor"]

Components

Data flow

CodeQL produces code scanning alerts over selected repositories. The taskflow agent reads YAML task definitions, fetches alert and source context using basic tools, iterates through alerts with templated prompts, records audit notes to a database after each task, delegates deterministic code facts to MCP tools, asks the LLM to reason about exploitability and false-positive checks, and emits a report or GitHub issue containing the verdict and supporting source references for human verification.

Why this architecture

The team used taskflows rather than a single prompt because complex security analysis can skip steps and exceed context windows. The database-backed note model makes failures resumable and debugging granular. MCP tools handle deterministic code facts because LLMs were inconsistent at extracting triggers and privilege context, while LLMs remain useful for human-like fuzzy reasoning over access checks, sanitizers, and exploitability.

Models used: Claude Sonnet 3.5 · LLM taskflows · MCP · CodeQL · GitHub Issues

Scale

• The taskflows triaged a large number of code scanning alerts.
  “we quickly triaged a large number of code scanning alerts”article, stated
• The reported use discovered about 30 real-world vulnerabilities since August.
  “discovered many (~30) real-world vulnerabilities since August”article, stated
• Taskflows support batch for-loop-style asynchronous execution over alerts.
  “can also perform a batch “for loop”-style task asynchronously.”article, stated
• The team tested a few CodeQL rules over a couple of months using mostly Claude Sonnet 3.5.
  “Over the course of a couple of months, we’ve tested our taskflows with a few CodeQL rules using mostly Claude Sonnet 3.5.”article, stated

Alternatives considered (roads not taken)

One large prompt for the whole triage workflow

Rejected because: Multi-step security tasks are often incomplete in one prompt and hard to debug.

Ask the LLM to gather deterministic workflow facts

Rejected because: The LLM inconsistently recorded workflow trigger events and privilege conclusions.

End-to-end autonomous exploit validation

Rejected because: The team only generates bug reports for human verification and does not give the LLM runtime environments to validate exploits.

Patterns

prompt-chaining (workflow)

Security triage is decomposed into YAML-defined tasks rather than one broad prompt.

tool-use-orchestration (agent architecture)

MCP servers perform deterministic file and code-fact tasks while the LLM handles semantic triage.

durable-execution (workflow)

Task notes are serialized after each step so the workflow can be debugged, reused, and resumed task by task.

human-in-the-loop-gate (security)

The agent produces reports with enough evidence for human auditors to verify and decide follow-up.

Tradeoffs

Generate reports instead of autonomous final judgments.

Chose Bug reports with details and conclusions over End-to-end exploit validation or fully automatic disclosure.

Rationale: Security decisions remain with human auditors while the agent removes repetitive triage work.

Use notes as a bag of information.

Chose Independent scoped tasks and serialized audit notes over Every task reading all prior context.

Rationale: Independent notes keep tasks focused and avoid distracting later tasks with unrelated context.

Limit the tools available during triage.

Chose Basic file fetching and searching plus MCP programmatic checks over Full static/dynamic analysis toolchain in the agent loop.

Rationale: The experiment isolates LLM taskflow triage on top of CodeQL alerts rather than giving the model broad execution powers.

Gotchas (what bit them)

LLMs are unreliable for deterministic code facts.

Root cause: Non-determinism caused missing trigger events and inconsistent privilege-context conclusions.

Resolution: Move those checks into MCP server tools and reserve the LLM for fuzzy semantic reasoning.

Static analyzers miss human-obvious access-control and sanitizer context.

Root cause: Checks come in many forms without easy code patterns; custom sanitizers and unreachable sources produce false positives.

Resolution: Use LLM tasks to reason about attacker reachability, authentication checks, sanitizers, and exploitability indicators.

Evaluation

The team evaluated taskflows by applying them to selected CodeQL rules over open source repositories, checking whether they removed false positives, generated human-verifiable reports, and led to confirmed real vulnerabilities. They also improved prompts based on observed false-positive patterns.

Metrics: real vulnerabilities discovered · false positives removed · human-verifiable report completeness · CodeQL rules tested · task consistency after MCP delegation

Outcomes

• GitHub Security Lab triaged a large number of CodeQL alerts using taskflows.
• The process discovered about 30 real-world vulnerabilities since August.
• Many discovered vulnerabilities were already fixed and published at article time.
• The framework and taskflows are open source.
• MCP delegation improved consistency for deterministic code checks.

The disclosed scale is a large number of code scanning alerts, a few CodeQL rules over a couple of months, and approximately 30 vulnerabilities found since August.