AI-Powered Incident Response System with Multi-Agent Investigation

incident.io · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Incident response and site reliability engineering

Incident.io needed an AI SRE product that could investigate production incidents quickly enough to help responders during the incident, while avoiding noisy, overconfident, or irrelevant recommendations in a high-pressure Slack-centered workflow.

Constraints:

• incident responders need useful context within minutes, not after a long offline analysis
• relevant evidence spans GitHub/GitLab, Slack, historical incidents, logs, metrics, traces, observability tools, and customer integrations
• the system must avoid positivity bias when an incident has too little information
• Slack incident channels are already noisy and cannot be flooded with every finding
• false positives and misleading confidence can send responders down the wrong path
• post-incident evaluation must capture partial value and useful negative findings, not only final root cause accuracy

Architecture

Incident.io built AI SRE around an investigation object that lives alongside each incident. An orchestration layer first gates whether there is enough information to investigate, then runs parallel searcher checks over code changes, Slack, historical incidents, and observability data. Search results become evidence-backed findings, findings become hypotheses, and the system critiques those hypotheses by generating questions. Specialized sub-agents answer those questions by drilling into metrics, traces, documentation, or other tools, then report back to the main reasoning loop. Results are progressively disclosed through Slack: a compact report in-channel, reasoning in the thread, and a richer dashboard with traces and citations. Evaluation uses post-incident time travel: after the incident closes, graders compare the AI investigation against what responders did and what actually caused the incident.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  incident-record["incident-record"] --> investigation-orchestrator["investigation-orchestrator"]
  investigation-orchestrator --> context-sufficiency-gate["context-sufficiency-gate"]
  investigation-orchestrator --> searcher-checks["searcher-checks"]
  searcher-checks --> github-gitlab-search["github-gitlab-search"]
  searcher-checks --> slack-search["slack-search"]
  searcher-checks --> historical-incident-search["historical-incident-search"]
  searcher-checks --> observability-search["observability-search"]
  searcher-checks --> retrieval-indexes["retrieval-indexes"]
  retrieval-indexes --> reranking-pipeline["reranking-pipeline"]
  searcher-checks --> findings-store["findings-store"]
  findings-store --> hypothesis-builder["hypothesis-builder"]
  hypothesis-builder --> critique-question-generator["critique-question-generator"]
  critique-question-generator --> sub-agent-workers["sub-agent-workers"]
  sub-agent-workers --> findings-store
  investigation-orchestrator --> slack-report-surface["slack-report-surface"]
  slack-report-surface --> reasoning-dashboard["reasoning-dashboard"]
  slack-report-surface --> ambient-slack-monitor["ambient-slack-monitor"]
  slack-report-surface --> user-feedback-loop["user-feedback-loop"]
  investigation-orchestrator --> time-travel-evaluator["time-travel-evaluator"]

Components

Data flow

An alert or incident opens an AI investigation. The system waits until it has enough context, then launches searcher checks across connected systems such as GitHub/GitLab, Slack, historical incidents, and Datadog/Grafana logs, metrics, and traces. Retrieval first narrows candidates through text similarity, summaries, tags, selective embeddings, and reranking, then the investigation layer converts useful hits into findings. The main loop builds hypotheses from those findings, critiques uncertainty, fans out specific questions to sub-agents, and folds the answers back into the hypothesis. A compact Slack report, thread reasoning, and dashboard citations expose the result to responders. After the incident closes, graders and user feedback score the investigation and feed prompt, retrieval, and integration improvements.

Why this architecture

Incident response demands fast, inspectable, and conservative assistance. Incident.io therefore decomposes the work into deterministic search/indexing where possible, LLM reranking where semantic judgment is useful, a gated multi-agent investigation loop for complex reasoning, and progressive disclosure in Slack to reduce noise. The post-incident evaluator exists because live incidents lack immediate ground truth, but resolved incidents provide the complete context needed to grade what the AI claimed and what responders used.

Models used: frontier models · GPT-4o · embedding models · large language models

Scale

• The company had been building incident-management products for about four years.
  “So, Incident IO, we started just over four years ago.”transcript, stated
• The product was designed for customers ranging from startups to Netflix-scale organizations.
  “So, we have customers like Lovable all the way up to Netflix and we help them get the right people in the room, help them know what to do, have a consistent process, hopefully make that situation a little bit less stressful and save them some time doing it.”transcript, stated
• The target investigation response time is about one or two minutes.
  “it should hopefully do that in about one or two minutes where a human at this point has probably actually just woken up”transcript, stated
• Early prototypes had to work around GPT-4o-era 25,000-token context windows.
  “especially back in days of GPC 40 with I know 25,000 token context windows or something like that. That just does not fit.”transcript, stated
• The team later discussed 200k-token context windows but still treated context management as necessary.
  “there's still context rot and like the more we can manage that context the better. But I feel like it wasn't very long ago we had the 200k context windows.”transcript, stated

Alternatives considered (roads not taken)

Run the full investigation immediately for every incident

Rejected because: The system first checks whether enough information exists; otherwise positivity bias would produce low-quality output from vague incidents.

Rely primarily on vector embeddings for retrieval

Rejected because: Vectors were difficult to debug and suffered from chunking, fuzzy matching, and versioning problems, so the team shifted toward summaries, tags, text similarity, and LLM reranking.

Put all investigation detail directly into Slack

Rejected because: Incident channels are noisy, so the product uses a compact report, thread details, and a dashboard for deeper citations.

Ask responders for feedback during the incident

Rejected because: Active incidents are urgent and distracting responders with feedback requests would likely annoy them, so Incident.io waits until after resolution.

Evaluate only the final root-cause answer

Rejected because: The team recognized that useful investigation includes ruling things out, surfacing relevant context, and providing value along the path to the cause.

Patterns

multi-agent-orchestration (architecture)

The investigation orchestrator coordinates parallel searcher checks, hypothesis critique, and sub-agents that answer focused questions before returning to the main reasoning loop.

rag-retrieval (retrieval)

The system searches code changes, Slack messages, historical incidents, and observability data to ground incident hypotheses in connected operational context.

retrieval-ranking (retrieval)

Incident.io builds broad candidate lists and reranks them with frontier models in batches, using top-k shortlists before final synthesis.

tool-use-orchestration (operations)

Sub-agents and searcher checks call operational tools and data sources such as metrics, traces, Slack, GitHub/GitLab, Kubernetes docs, Datadog, and Grafana.

confidence-thresholding (evidence)

The system critiques hypotheses and calibrates the assertiveness of reports based on confidence and uncertainty.

eval-driven-development (evaluation)

The team evaluates after incident closure, scores investigation claims, and uses component-level graders plus prompt eval suites to debug failures.

context-compaction (context)

Large code changes and incident artifacts are summarized, tagged, shortlisted, and reranked so the reasoning loop receives relevant context instead of raw volumes of data.

human-in-the-loop-gate (control)

The system works in Slack with responders, exposes citations and uncertainty, and collects human feedback rather than autonomously fixing production incidents.

observability-tracing (operations)

Incident investigations include connections into logs, metrics, traces, Datadog, and Grafana so hypotheses can be tested against operational signals.

data-quality-loop (learning)

Post-incident evaluation and missing-integration analysis reveal which data sources or permissions would have improved investigations.

Tradeoffs

Gate investigation start based on available context

Chose wait until the incident has enough information over force an investigation from a vague incident stub.

Rationale: Gating avoids positivity-bias output and keeps investigation quality higher when responders most need trustworthy help.

Use hybrid deterministic retrieval plus LLM reranking

Chose text summaries, tags, conventional Postgres indexing, selective embeddings, and reranking over heavy reliance on pure vector similarity.

Rationale: Hybrid retrieval preserves semantic ranking where it helps while making failed searches easier to inspect and debug.

Progressively disclose incident findings

Chose compact Slack report, thread reasoning, and dashboard citations over putting all findings directly into the incident channel.

Rationale: Responders need immediate signal without losing the ability to inspect evidence and reasoning when they have time.

Evaluate after incident closure

Chose time-travel scorecards and graders after the incident has resolved over requiring live ground truth or only human-labeled evaluations.

Rationale: Resolved incidents naturally reveal the cause, actions, and responder behavior needed to score investigation usefulness at scale.

Let agents handle repeat monitoring tasks

Chose agent callback tasks that report only when something changes over having humans refresh status pages or metrics repeatedly.

Rationale: AI agents are well-suited for repetitive monitoring work that is valuable but tedious for responders during incidents.

Gotchas (what bit them)

Vague incident reports can trigger positivity-bias output if the investigation starts too early.

Root cause: LLMs tend to produce requested output even when the available incident context is insufficient.

Resolution: Incident.io gates the full investigation until the system knows enough about the nature of the problem.

Pure vector retrieval was hard to debug under incident-response pressure.

Root cause: Embedding matches, chunking behavior, and vector similarity failures are less inspectable than tags and text summaries.

Resolution: The team reduced reliance on vectors and moved toward text summaries, tagged keywords, conventional indexing, and model reranking.

Large context windows did not remove context-management work.

Root cause: Incident evidence can include huge code changes and many operational artifacts; long context windows still suffer from context rot.

Resolution: Incident.io preprocesses, summarizes, tags, and reranks candidate context before reasoning over it.

Technically real findings can still be harmful if they are irrelevant.

Root cause: Modern frontier models hallucinate less often, but false positives and misleading relevance can distract responders during critical incidents.

Resolution: The team tunes for signal over noise, confidence-calibrated language, automated evals, and feedback loops.

Prompts can overfit to the builder's own operational context.

Root cause: Engineers may encode assumptions from Incident.io's internal practices that do not hold for customer environments.

Resolution: The team explicitly watches for internal-context leakage while debugging evals and adjusting prompts.

Live user feedback can itself become incident-response noise.

Root cause: Responders are under pressure and may be annoyed by feedback prompts during the incident.

Resolution: Incident.io delays formal feedback until after the incident while still allowing in-channel negative feedback if responders choose to give it.

Evaluation

Incident.io evaluates AI SRE through post-incident time-travel scorecards, nuanced false-positive/true-positive style metrics, component-specific graders, prompt-level eval suites, user clicks and feedback, and delayed Slack usefulness checks after the incident has ended.

Metrics: scorecard of investigation claims after incident closure · false positives, true positives, false negatives, and true negatives · signal-to-noise ratio of final claims · root-cause-analysis quality · RAG search precision and recall · responder clicks and viewed findings · post-incident usefulness feedback

Outcomes

• Incident.io describes AI SRE as able to investigate across code, Slack, historical incidents, logs, metrics, and traces and return hypotheses within about one or two minutes.
• The team moved away from hard-to-debug vector-first retrieval toward summaries, tags, deterministic indexing, and LLM reranking.
• The product exposes compact Slack reports, threaded reasoning, dashboard traces, citations, confidence, and conversational follow-up rather than a single opaque answer.
• The evaluation system can score investigations after incident closure without requiring humans to manually create ground truth for every case.
• The team identified false positives, prompt overfitting to internal context, missing integrations, and feedback timing as production concerns to manage continuously.

The transcript describes Incident.io as about four years old, serving customers from startups to Netflix, building AI for about 1.5 years, targeting one-to-two-minute incident investigation output, moving from 25k-token-era constraints to larger 200k-token contexts while still managing context, and evaluating after every incident closes.