Agent-Driven Development for AI Research Using GitHub Copilot CLI

github · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI research tooling

GitHub Copilot researchers needed to analyze large agent-trajectory benchmark outputs without spending human time on repetitive intellectual toil.

Constraints:

Hundreds of thousands of lines of trajectory code
Shared researcher workflows
Agent-built contributions needed maintainability

Architecture

GitHub's eval-agents workflow uses Copilot CLI and the Copilot SDK as an agent-development platform for applied-science analysis. Benchmark trajectories from TerminalBench2, SWEBench-Pro, and internal runs become analysis inputs; Copilot CLI with Claude Opus 4.6 helps build specialized agents, tools, MCP integrations, and skills; researchers use planning-mode prompts before implementation; repository architecture, strict typing, linters, contract tests, and CI/CD guardrails protect the agent-built system. The result is a maintainable tool for automating trajectory analysis rather than a one-off script.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  benchmark-trajectories[Benchmark trajectories] --> eval-agents-tool[Eval-agents tool]
  eval-agents-tool --> copilot-cli[Copilot CLI]
  copilot-cli --> claude-opus-4-6[Claude Opus 4.6]
  copilot-cli --> copilot-sdk[Copilot SDK]
  copilot-sdk --> tools-mcp-and-skills[Tools MCP and skills]
  planning-prompts[Planning prompts] --> copilot-cli
  guardrail-suite[Guardrail suite] --> eval-agents-tool
  eval-agents-tool --> researcher-workflow[Researcher workflow]
  planning-prompts --> researcher-workflow

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Benchmark trajectories`	TerminalBench2, SWEBench-Pro, and other agent-run JSON traces requiring analysis.	`Eval-agents tool`
`Eval-agents tool`	Research tool that automates analysis of agent trajectories.	`Benchmark trajectories`, `Copilot CLI`, `Researcher workflow`
`Copilot CLI`	Coding agent interface used to build and extend eval-agents.	`Eval-agents tool`, `Claude Opus 4.6`, `Copilot SDK`
`Claude Opus 4.6`	Model used by the coding agent.	`Copilot CLI`
`Copilot SDK`	SDK layer exposing tools, MCP servers, skills, and agentic primitives.	`Copilot CLI`, `Tools MCP and skills`
`Tools MCP and skills`	Reusable tool and skill layer used by eval-agents and coding agents.	`Copilot SDK`
`Planning prompts`	Verbose planning-mode prompts used before implementation.	`Copilot CLI`, `Researcher workflow`
`Guardrail suite`	Strict typing, linters, integration tests, contract tests, and CI/CD checks.	`Eval-agents tool`
`Researcher workflow`	Applied-science team workflow for creating agents and analyzing benchmark behavior.	`Eval-agents tool`, `Planning prompts`

Data flow

Evaluation benchmark runs produce JSON trajectories with agent thoughts and actions. Researchers build and run eval-agents through Copilot CLI and the Copilot SDK, using tools, MCP servers, and skills to analyze those trajectories. Planning prompts define work, the agent writes code and tests, guardrails catch regressions, and the resulting tool enables the Copilot Applied Science team to automate repeated analysis.

3 supporting quotes

“trajectories, which are essentially lists of the thought processes and actions agents take while performing tasks.”article, stated

“* Coding agent: Copilot CLI * Model used: Claude Opus 4.6 * IDE: VSCode It’s also noteworthy that I leveraged the Copilot SDK to accelerate agent creation, which is powered under the hood by the Copilot CLI.”article, stated

“This gave me access to existing tools and MCP servers, a way to register new tools and skills, and a whole bunch of other agentic goodness out of the box that I didn’t have to reinvent myself.”article, stated

Why this architecture

The author chose Copilot CLI and the SDK because they supplied agentic building blocks, tools, MCP servers, and skill registration without rebuilding the harness. The repository process intentionally treats the coding agent like a junior engineer: give it rich context, planning time, tests, and CI guardrails so rapid iteration does not become unreviewable churn.

3 supporting quotes

“agents work best when you’re conversational, verbose, and when you leverage planning modes before agent modes.”article, stated

“When Copilot has these tools available in its development loop, it can check its own work. You’re setting it up for success, much in the same way you’d set up a junior engineer for success in your project.”article, stated

Models used: Claude Opus 4.6

Scale

Trajectory analysis can involve hundreds of thousands of lines of code-like JSON.
“Multiply that over dozens of tasks in a benchmark set and again over the many benchmark runs needing analysis on any given day, and we’re talking hundreds of thousands of lines of code to analyze. It’s an impossible task to do alone, so I would typically turn to AI to help.”article, stated
Five researchers created 11 agents, four skills, and eval-agent workflows in under three days.
“We had five folks jump into the project for the first time, and we created a total of 11 new agents, four new skills, and the concept of eval-agent workflows (think scientist streams of reasoning) in less than three days.”article, stated
The initial sprint changed 28,858 added and 2,884 removed lines across 345 files.
“That amounted to a change of +28,858/-2,884 lines of code across 345 files. Holy crap!”article, stated

Alternatives considered (roads not taken)

Manually pore through benchmark trajectories

Rejected because: The repetitive intellectual analysis was too large and automatable.

“However, the engineer in me saw this repetitive task and said, “I want to automate that.” Agents provide us with the means to automate this kind of intellectual work, and thus eval-agents was born.”article, stated

Give terse implementation prompts directly to the agent

Rejected because: Planning-mode conversation produced better architecture and guardrails for complex tasks.

“I found it far more effective to put some stream-of-consciousness musings about a problem I was chewing on into a prompt and working with Copilot in planning mode than to give it a terse problem statement or solution.”article, stated

Trust agent outputs without reinforcing process

Rejected because: The author adds tests, prompts, linters, and CI/CD guardrails when mistakes occur.

“That means we add processes and guardrails to help prevent the agent from making mistakes, but when it does make a mistake, we add additional guardrails and processes—like more robust tests and better prompts—so the agent can’t make the same mistake again.”article, stated

Patterns

`tool-use-orchestration` (development)

The agent-driven tool used Copilot CLI, SDK capabilities, tools, MCP servers, and skills.

`prompt-versioning` (development-practice)

The article emphasizes prompting strategies and planning modes as part of repeatable agent-driven development.

“* Prompting strategies: agents work best when you’re conversational, verbose, and when you leverage planning modes before agent modes.”article, stated

Tradeoffs

Build process guardrails around agents instead of blaming individual agent mistakes.

Chose More robust tests, better prompts, typing, linters, and CI/CD principles over Trusting agent outputs without reinforcing the development loop.

Rationale: The author treats agent-driven work like human team work: mistakes trigger stronger process rather than one-off blame.

“Applying this same philosophy to agent-driven development has been fundamental to unlocking this incredibly rapid iteration pipeline. That means we add processes and guardrails to help prevent the agent from making mistakes”article, stated

How much structure to give coding agents.

Chose planning prompts plus guardrails over terse requests and unchecked agent changes.

Rationale: Complex research tooling needed senior-engineer style guidance and process reinforcement.

“need handholding for the more complex problems you’d only entrust to your more senior engineers.”article, stated

Gotchas (what bit them)

Coding agents still need senior-engineer-style guidance for complex work.

Root cause: Agents are strong on well-scoped tasks but require handholding for ambiguous, architectural, or high-risk changes.

Resolution: The author used verbose planning prompts, assumptions, research, and contract-test guardrails before letting Copilot implement.

“We know that AI coding agents are really good at solving well-scoped problems but need handholding for the more complex problems you’d only entrust to your more senior engineers.”article, stated

Evaluation

The tool exists to analyze standardized agent benchmark trajectories such as TerminalBench2 and SWEBench-Pro.

Metrics: benchmark trajectories · agent performance analysis

“A large part of my job involves analyzing coding agent performance as measured against standardized evaluation benchmarks, like [TerminalBench2](https://www.”article, stated

Outcomes

Five researchers built 11 agents, four skills, and eval-agent workflows in under three days, according to the summary.

The work automated repetitive analysis of agent trajectory files.

“As an AI researcher, I recently took this beyond what was previously possible and have automated away my intellectual toil. And now I find myself maintaining this tool to enable all my peers on the Copilot Applied Science team to do the same.”article, stated