AI Agent System for Automated Security Investigation and Alert Triage

slack · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Security alert investigation

Slack security on-call needed to investigate alerts from billions of daily events without manual evidence gathering dominating shifts.

Constraints:

high-stakes security
multiple data sources
variable agent behavior
structured output reliability

Architecture

Slack’s system turns security alerts into queued investigations run by workers. A Hub exposes the API, persistence, metrics, and launch surface; workers execute structured model tasks; a Director agent plans and asks questions; domain Expert agents interrogate specific data sources exposed through MCP/tool calls; and a Critic reviews claims, tool calls, and tool results before the Director advances or finalizes the report. This creates a knowledge pyramid: expensive, token-heavy expert gathering stays low-level, while reviewed findings flow upward to higher-level decisions.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  detection-tools["detection-tools"]
  hub-api["hub-api"]
  investigation-worker["investigation-worker"]
  director-agent["director-agent"]
  expert-agents["expert-agents"]
  critic-agent["critic-agent"]
  mcp-data-sources["mcp-data-sources"]
  persistent-storage["persistent-storage"]
  structured-output-tasks["structured-output-tasks"]
  journaling-tool["journaling-tool"]
  detection-tools --> hub-api
  hub-api --> investigation-worker
  hub-api --> persistent-storage
  investigation-worker --> director-agent
  director-agent --> expert-agents
  director-agent --> journaling-tool
  expert-agents --> mcp-data-sources
  expert-agents --> critic-agent
  critic-agent --> director-agent
  structured-output-tasks --> director-agent
  structured-output-tasks --> expert-agents
  structured-output-tasks --> critic-agent

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`detection-tools`	Existing security detection systems that can trigger investigations from Slack’s event pipeline.	`hub-api`
`hub-api`	Service API, investigation launch surface, persistent-storage interface, and metrics endpoint for activity, token use, and cost.	`detection-tools`, `investigation-worker`, `persistent-storage`
`investigation-worker`	Executes queued investigations and model-task chains, emitting an event stream back through the service.	`hub-api`, `director-agent`
`director-agent`	Plans and progresses the investigation by forming questions for Experts and deciding the next step from reviewed findings.	`investigation-worker`, `expert-agents`, `critic-agent`, `journaling-tool`
`expert-agents`	Domain-specific agents with unique knowledge and data-source/tool access that produce findings.	`director-agent`, `mcp-data-sources`, `critic-agent`
`critic-agent`	Reviews expert findings and supporting tool traces, identifies credible findings, and assembles a timeline.	`expert-agents`, `director-agent`
`mcp-data-sources`	Subset of Slack data sources exposed safely through stdio MCP/tool interfaces.	`expert-agents`
`persistent-storage`	Stores investigations so they can be viewed, shared, queried, and used for reporting/metrics.	`hub-api`
`structured-output-tasks`	JSON-schema-bound model tasks that constrain each invocation to a single purpose and output structure.	`investigation-worker`, `director-agent`, `expert-agents`, `critic-agent`
`journaling-tool`	Investigation journal/memory tool used by the director agent to track investigation state and findings.	`director-agent`

Data flow

An alert or ad-hoc request enters the service through the Hub and becomes an investigation task. A worker picks it up and runs a chain of structured-output model invocations. The Director journals the plan and asks questions; Experts use their domain data sources and tools to generate findings; the Critic inspects claims and supporting tool calls/results, assembles a credible timeline, and sends that reviewed knowledge back to the Director, which decides the next step or produces the final report.

3 supporting quotes

“The Director agent poses a question and domain expert agents respond, generating findings. The Critic agent reviews findings for quality and assembles a timeline using the most credible. The Director uses the high-quality findings and timeline to determine how to progress the investigation.”article, stated

“Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed.”article, stated

“During the review process the Critic inspects the experts’ claims and the tool calls and tool results used to support them, which also incurs a significant token overhead.”article, stated

Why this architecture

Slack moved away from a large prompt because prompt guidelines did not give reliable investigation control. Chaining small structured tasks gave each model invocation a narrow purpose, and persona separation let Specialists gather evidence while a Critic checked quality. The service wrapper was needed because a coding-agent prototype could not support real-time observability, stored/shareable investigations, ad-hoc launches, metrics, cost management, or integration with existing detection tools.

3 supporting quotes

“ultimately prompts are just guidelines; they’re not an effective method for achieving fine-grained control.”article, stated

“Our solution was to break down the complex investigation process we’d described in the prompt of our prototype into a sequence of model invocations, each with a single, well-defined purpose and output structure.”article, stated

“We needed an interface that would let us observe investigations occurring in realtime, view and share past investigations, and launch ad-hoc investigations. Critically, we needed a way of integrating the system into our existing stack”article, stated

Models used: LLMs

Scale

Slack’s security event ingestion pipeline processes billions of events per day.
“Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources.”article, stated
The Hub exposes metrics specifically so the team can visualize system activity, token usage, and cost.
“the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost.”article, stated
The system explicitly manages token-intensive work by pushing broad data gathering to the base of a knowledge pyramid.
“At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive.”article, stated

Alternatives considered (roads not taken)

A single large prompt with methodology and formatting instructions

Rejected because: It produced impressive results sometimes, but could jump to convenient/spurious conclusions and prompt refinements did not provide consistent fine-grained control.

“sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process.”article, stated

A coding agent CLI as the production execution harness

Rejected because: It was useful for prototyping but lacked the API, persistence, real-time observability, sharing, ad-hoc launch, and detection-tool integration required for production.

“Our prototype used a coding agent CLI as an execution harness, but that wasn’t suitable for a practical implementation. We needed an interface that would let us observe investigations occurring in realtime, view and share past investigations, and launch ad-hoc investigations.”article, stated

Patterns

`multi-agent-orchestration` (security)

Director, expert, and critic agents coordinate investigation questions, findings, timelines, and next steps.

“The Director agent poses a question and domain expert agents respond, generating findings.”article, stated

`structured-output-contract` (quality)

Slack uses JSON-schema structured outputs to turn vague instructions like evidence questioning into explicit tasks.

“Each task was given a structured output format.”article, stated

Tradeoffs

How to control investigation behavior.

Chose application-orchestrated structured tasks over one broad autonomous prompt.

Rationale: More scaffolding gives predictable behavior but requires explicit task design and context propagation.

“For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process.”article, implied

How to allocate model work across cost tiers during investigations.

Chose Use a knowledge pyramid where token-heavy expert gathering feeds reviewed findings upward to higher-level decision making. over Running every investigation step with the same expensive model context and full raw evidence..

Rationale: The architecture concentrates costly tool-heavy analysis in expert agents and lets reviewed findings flow to the Director, trading complexity for cost and controllability.

“At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive.”article, stated

“Once the Critic has completed its review, it assembles an up to date investigation timeline, integrating the running investigation timeline and newly gathered findings into a coherent narrative.”article, stated

Gotchas (what bit them)

A single prompt produced highly variable investigations.

Root cause: The prototype could either cross-reference evidence well or jump to spurious conclusions without enough self-questioning.

Resolution: Decompose investigation into structured agent/persona tasks and add a Critic review role.

“However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods.”article, stated

Structured outputs improved control but did not eliminate model cheating or hallucination.

Root cause: The schema constrains the final shape, but the model can still produce low-quality reasoning inside that shape.

Resolution: Make evidence-questioning a separate investigation task and keep critic review in the workflow.

“Structured outputs are also subject to the usual problems of cheating and hallucination.”article, stated

Evaluation

Slack emphasizes predictable structured behavior and verifiable reports; no public aggregate metric is given in the excerpt.

Metrics: structured task success · finding quality · timeline credibility

“In our initial prototype, we included guidance to “question your evidence”, but had mixed success.”article, implied

Outcomes

The system changed on-call work from manual evidence gathering toward supervising agent-generated reports.

The source also notes emergent discoveries, but this extraction keeps claims close to the article excerpts.

“Results include transformed on-call workflows from manual evidence gathering to supervision of agent teams, interactive verifiable reports, and emergent discovery capabilities where agents spontaneously identified security issues beyond the original alert scope”summary, stated