AI Agents for Interpretability Research: Experimenter Agents in Production

goodfire · Research & Academia · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI interpretability research

Goodfire needed agents that could run valid experiments and discover insights, not just produce durable software artifacts.

Constraints:

• scientific validity
• notebook interactivity
• context engineering
• reward hacking risk

Architecture

Goodfire’s Scribe setup treats interpretability research as stateful, auditable experimentation rather than ordinary coding. Experimenter agents use a Jupyter server/notebook MCP to execute code through Jupyter messaging, inspect text/errors/images, and iterate cell-by-cell while retaining loaded models and variables. Notebooks become timestamped, labeled experiment artifacts. Because research lacks a simple test-suite reward, the workflow adds human verification, post-hoc critic agents, live critic-in-the-loop hooks, critic copilots, context engineering, and ensemble/parallel experiment strategies.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  scribe-experimenter-agent["scribe-experimenter-agent"]
  notebook-mcp["notebook-mcp"]
  jupyter-kernel["jupyter-kernel"]
  notebook-artifact["notebook-artifact"]
  critic-agents["critic-agents"]
  context-engineering-layer["context-engineering-layer"]
  researcher-human-review["researcher-human-review"]
  interpretability-task-suite["interpretability-task-suite"]
  scribe-experimenter-agent --> notebook-mcp
  notebook-mcp --> jupyter-kernel
  jupyter-kernel --> notebook-artifact
  notebook-artifact --> critic-agents
  critic-agents --> researcher-human-review
  context-engineering-layer --> scribe-experimenter-agent
  interpretability-task-suite --> scribe-experimenter-agent
  notebook-artifact --> researcher-human-review

Components

Data flow

A researcher gives Scribe an open-ended experiment and relevant setup. The agent uses notebook MCP access to run code interactively, keeping packages, variables, and loaded models alive across cells. Outputs, plots, markdown, and code accumulate in a timestamped notebook artifact. After or during the run, critic agents/c copilots inspect results and transcripts for errors, methodological flaws, shortcutting, p-hacking, or over-optimistic conclusions; humans verify the research claim before treating it as knowledge.

Why this architecture

Goodfire separates experimenter agents from developer agents because research outputs are claims, not just compiling code. Notebooks improve iteration by preserving expensive setup and letting agents rerun one cell instead of whole scripts. Critic and human-review layers are necessary because research has no universal unit-test reward, agents are biased toward optimistic shortcutting/p-hacking/eureka narratives, and much useful context is non-linear or tacit.

Models used: AI agents · Scribe · GPT-5 critic copilot · Claude Code · Codex · Gemini CLI

Scale

• Goodfire reports every researcher and engineer uses AI tools daily after building internal agentic infrastructure.
  “At Goodfire, every one of our researchers and engineers leverages AI tools daily.”article, stated
• Human verification is the main bottleneck to scaling experiments performed by research agents.
  “human verification is the main bottleneck to scaling experiments performed by research agents”article, stated
• The notebook MCP package is open-sourced with adapters for Claude Code, Codex, and Gemini CLI.
  “We are open sourcing a reference implementation of the Jupyter server/notebook MCP system that can be used with experimenter agents like Scribe,”article, stated
• The task suite includes circuit discovery, universal neuron detection, and linear probe analysis as directional interpretability tasks.
  “including problems like circuit discovery, universal neuron detection, and linear probe analysis — that can be run using Scribe.”article, stated

Alternatives considered (roads not taken)

Treat experimenter agents like ordinary developer agents

Rejected because: Research tasks need different context, tools, and validation because their output is a conclusion rather than a code artifact.

Script-only execution instead of stateful notebook access

Rejected because: Scripts force agents to rerun setup after small errors, reducing iteration speed and experimental throughput.

Rely on the experimenter agent’s own final report without critique

Rejected because: Agents can shortcut, p-hack, and treat bugs as breakthroughs, so Goodfire uses critics and human validation.

Patterns

tool-use-orchestration (research)

Experimenter agents receive interactive notebook access so they can run and iterate on scientific experiments.

eval-driven-development (quality)

Goodfire explicitly separates experiment validity from software artifact creation and adds validation/critic mitigations.

Tradeoffs

How much autonomy to allow in research.

Chose agent-run experiments with human validation over manual-only exploration or unchecked autonomous claims.

Rationale: Agents accelerate experimentation, but conclusions must be validated to avoid shallow or reward-hacked results.

How to validate research-agent outputs without a unit-test-like reward.

Chose Use human verification plus critic agents/copilots around the notebook artifact. over Trusting the experimenter agent’s own final interpretation of results..

Rationale: Goodfire says research lacks a clear verifiable reward and agents are biased toward optimism, so external critique and human judgment are needed.

Gotchas (what bit them)

Developer-agent assumptions do not transfer cleanly to scientific experimentation.

Root cause: Research agents need valid conclusions and discovery behavior, while most agent tooling/training/evals optimize software tasks.

Resolution: Design experimenter-specific tools, validation habits, and critic systems instead of reusing coding-agent defaults blindly.

The broader agent ecosystem is biased toward software development tasks.

Root cause: Training data, demand, and benchmarks are richer for coding agents than for agents that run research experiments.

Resolution: Build experimenter-specific tools and evals instead of assuming coding-agent benchmarks capture research quality.

Evaluation

Goodfire reports representative experiments and lessons, with validation challenges treated as first-order.

Metrics: experiment validity · traceability · critic review

Outcomes

• Scribe agents helped run interpretability experiments in genomics, vision, diffusion, and circuit analysis.

The article is candid about both successes and remaining limitations.