Using AI to Debug and Manage Complex AI Systems in Production

incident · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI incident investigation and debugging

Incident needed to debug and improve AI investigation systems whose agent, prompt, tool, and telemetry traces had become too large and nested for humans to inspect manually at production scale.

Constraints:

• The production AI system involved dozens of agents, hundreds of prompts, and thousands of tool calls.
• A single investigation could require hundreds of telemetry queries and an hour or more of human review.
• The system ran investigations daily across hundreds of customer accounts.
• Coding agents could not directly work with multi-megabyte YAML eval files or web debugging UIs.
• Debugging had to connect traces, evals, code, and production fixes.

Architecture

Incident built AI-native internal tooling for AI-system debugging. Evals live as YAML beside Go code but are manipulated through an agent-friendly CLI and runbooks. Complex trace UIs can be exported as structured file systems that Claude Code can grep, navigate, and analyze. For scale, back tests download thousands of investigations into file trees, Scrapbook playbooks launch parallel agents, cluster failure cohorts, and connect trace findings back to code and eval-driven fixes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  production-investigation-system["Production investigation system"] --> telemetry-sources["Telemetry sources"]
  production-investigation-system --> prompt-and-agent-hierarchy["Prompt and agent hierarchy"]
  production-investigation-system --> back-test-runner["Back test runner"]
  prompt-and-agent-hierarchy --> file-system-trace-export["File system trace export"]
  eval-yaml-suite["Eval YAML suite"] --> eval-cli["Eval CLI"]
  eval-cli --> coding-agent-runbooks["Coding agent runbooks"]
  file-system-trace-export --> coding-agent-runbooks
  back-test-runner --> scrapbook-analysis-pipeline["Scrapbook analysis pipeline"]
  file-system-trace-export --> scrapbook-analysis-pipeline
  coding-agent-runbooks --> scrapbook-analysis-pipeline

Components

Data flow

A problematic production interaction or back-test batch is captured into eval YAML or a downloadable file system. A coding agent uses CLI commands and markdown runbooks to inspect prompts, inputs, outputs, traces, and source code; adds an eval proving the failure; changes the relevant prompt or code; runs regression checks; and, for back tests, aggregates investigation analyses into cohorts that point developers at recurring failure modes and candidate fixes.

Why this architecture

Incident made debugging artifacts file- and CLI-native because coding agents are much better at navigating textual file systems and command surfaces than web UIs or giant YAML blobs. The same representation supports individual trace debugging and large back-test analysis.

Models used: Claude Code · AI grading models · Coding agents

Scale

• Incident's AI capabilities had been built for approximately 18-24 months.
  “has been building these AI capabilities for approximately 18-24 months.”summary, stated
• The system queries hundreds of telemetry sources.
  “automatically runs comprehensive investigations that query hundreds of telemetry sources including logs, metrics, traces, and historical incident data”summary, stated
• A human review of one investigation could take an hour or more.
  “often taking an hour or more of human analysis to review the incident details, postmortem documentation”summary, stated
• The architecture included at least 10 agents and 50 or more components.
  “the system architecture includes at least 10 different agents with 50 or more components in the hierarchy powering interactions.”summary, stated
• Incident runs thousands of investigations daily across hundreds of customer accounts.
  “They run thousands of investigations daily across hundreds of customer accounts through what they call "back tests,"”summary, stated
• Scrapbook starts approximately 25 agents simultaneously for analysis.
  “they parallelize analysis by starting approximately 25 agents simultaneously, each building independent analysis of individual investigations.”summary, stated

Alternatives considered (roads not taken)

Let humans inspect all production investigation traces manually

Rejected because: Trace volume and complexity made manual analysis intractable.

Expose only web debugging UIs

Rejected because: Agents work better with structured file systems than with UI-only traces.

Use aggregate back-test metrics alone

Rejected because: High-level accuracy numbers did not explain why performance changed or how to fix customer-specific failures.

Patterns

eval-driven-development (evaluation)

Incident treats evals as unit tests and uses agents to create failing evals, modify prompts, run regressions, and validate fixes.

tool-use-orchestration (agent tooling)

Coding agents use CLI tools, file exports, eval commands, code navigation, and analysis runbooks as a debugging toolchain.

multi-agent-orchestration (analysis)

Scrapbook runs many agents in parallel over investigation batches and then clusters their findings.

context-engineering (agent context)

Incident converts UI state, prompts, inputs, outputs, traces, and code context into navigable file systems that agents can inspect with standard tools.

durable-execution (analysis pipeline)

Long-running Scrapbook analyses persist intermediate files so they can resume if interrupted.

Tradeoffs

Agent-native file exports

Chose Download traces and UI content as structured file systems over Keep trace inspection only inside web UIs.

Rationale: File systems let coding agents use grep, navigation, code context, and standard tools for systematic debugging.

Runbook-driven agent workflow

Chose Codify debugging and eval-modification workflows as skills/runbooks over Ask agents to improvise debugging steps.

Rationale: Runbooks constrain the agent loop from failure reproduction to prompt change and regression checks.

Parallel agent back-test analysis

Chose Launch about 25 agents over individual investigations, then cluster findings over Have humans manually inspect thousands of investigations.

Rationale: Parallel analysis made high-volume investigation review feasible and produced actionable cohorts.

Gotchas (what bit them)

Realistic production evals became too large for agents to edit.

Root cause: Captured incident contexts produced multi-megabyte YAML files that exceeded useful context limits.

Resolution: Incident built an eval CLI with focused commands so agents could manipulate the suite without loading whole files.

Debugging UIs blocked systematic agent analysis.

Root cause: Web UIs were designed for human inspection, not agent navigation across large trace sets.

Resolution: Incident made UI content downloadable as structured file systems.

Aggregate back-test metrics hid root causes.

Root cause: Overall accuracy numbers did not explain customer-specific or cohort-level failure patterns.

Resolution: Scrapbook analyzes individual investigations in parallel, then clusters failure cohorts.

AI debugging tools can introduce recursive complexity.

Root cause: Meta-level AI tooling may itself require debugging and maintenance.

Resolution: Organizations adopting the pattern need to account for extra layers of tooling complexity and dependency risk.

Evaluation

Incident evaluates its AI investigation system through AI-unit-test evals, repeatability checks, back tests across accounts, aggregate RCA metrics, cohort clustering, and production validation of fixes.

Metrics: Pass/fail eval status · Repeatability of eval outcomes · Back-test RCA accuracy such as 86% · Cohort-level failure patterns · Production validation after deployment

Outcomes

• Developers can ask coding agents to inspect problematic AI interactions and identify which system component should change.
• Back-test analysis can discover cohorts of investigation failures rather than only aggregate scores.
• Incident used the loop to generate a pull request and deploy a production fix.
• The approach saved days or weeks of engineering time for complex analysis according to the speaker.

The system supports AI investigations across hundreds of customer accounts, thousands of daily investigations, at least 10 agents, 50+ hierarchy components, and parallel analysis with about 25 agents.