Managing Context in Long-Run Agentic Security Investigation Systems

slack · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Long-running agentic security investigations

Slack needed multi-agent security investigations to stay coherent across long-running investigations without overflowing context windows or letting unsupported findings propagate.

Constraints:

• Security investigations can span hundreds of inference requests and generate megabytes of output.
• Language model APIs are stateless, so naive message-history accumulation fills context windows and degrades quality.
• Each agent needs a tailored view of investigation state; too little context disconnects the team, while too much context encourages confirmation bias.
• Expert findings can be invented, unsupported, or based on misinterpreted data, so security automation needs explicit validation.

Architecture

Slack built a multi-agent investigation system with a Director, specialist Experts, and a Critic, but replaced full message-history carryover with three maintained context channels: the Director's Journal, the Critic's Review, and the Critic's Timeline. The Journal supplies structured working memory, the Review scores and filters Expert findings using evidence-inspection tools, and the Timeline consolidates credible findings into a coherent chronology.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Director["Director agent"] --> Journal["Director's Journal"]
  Director --> Experts["Specialist Expert agents"]
  Experts --> Tools["Investigation tools"]
  Experts --> Critic["Critic agent"]
  Tools --> ReviewTools["Critic review tools"]
  ReviewTools --> Critic
  Critic --> Review["Critic's Review"]
  Review --> Timeline["Critic's Timeline"]
  Journal --> Timeline
  Timeline --> Director
  Journal --> Shared["Shared prompt context"]
  Review --> Shared
  Timeline --> Shared
  Shared --> Director
  Shared --> Experts
  Shared --> Critic

Components

Data flow

An investigation moves through phases and rounds until the Director concludes it. The Director asks specialist Experts to investigate with domain tools, records decisions and open questions in the Journal, and receives findings. The Critic reviews Expert reports with tools for inspecting cited calls, results, and available toolsets, assigns credibility scores, and then builds a Timeline from the latest Review, previous Timeline, and Journal. Future agent invocations receive these context channels instead of full message history.

• Director’s Journal: The Director’s structured working memory
• Critic’s Review: Annotated findings report with credibility scores
• Critic’s Timeline"

Why this architecture

Slack chose purpose-built context channels because generic message-history accumulation breaks down in long investigations. The design gives each agent enough shared orientation to stay aligned while avoiding full-history overload and confirmation bias. A stronger but bounded Critic role concentrates expensive model capacity on validation rather than every exploratory step.

Models used: LLMs · stronger model for Critic role

Scale

• Investigations can span hundreds of inference requests.
  “Complex security investigations can span hundreds of inference requests and generate megabytes of output, requiring special handling.”article, stated
• The system can generate megabytes of investigation output.
  “Complex security investigations can span hundreds of inference requests and generate megabytes of output, requiring special handling.”article, stated
• Slack measured the Review scoring distribution over 170,000 reviewed findings.
  “The following table shows the distribution of classifications over 170,000 reviewed findings. Slightly over a quarter of findings don’t meet the plausibility threshold.”article, stated
• One example investigation retrieved 6,046 session events.
  “Reviewed expert findings (6,046 events showing package install triggered event). All 4 experts agree: FALSE POSITIVE.”article, stated
• The example timeline reached 0.83 confidence.
  “Confidence Score: 0.83”article, stated

Alternatives considered (roads not taken)

Carry full message history forward between agent invocations

Rejected because: Accumulated messages increase costs and latency, degrade model quality, and eventually exceed context windows.

Give the Critic all raw investigation information directly

Rejected because: The number of findings makes direct inclusion impractical, so the Critic uses tools to inspect cited evidence on demand.

Use the same model quality uniformly across all agents

Rejected because: Slack concentrated a stronger model on the bounded Critic role where nuance interpretation matters and token volume is controlled.

Let Review scores alone determine final investigation state

Rejected because: Slack adds Timeline consolidation so findings must cohere with the full evidence chain before being preserved.

Patterns

multi-agent-orchestration (agent_architecture)

Slack decomposed investigations into Director orchestration, specialist Expert evidence gathering, and Critic validation/timeline roles.

context-engineering (context_management)

The system curates role-specific context through Journal, Review, and Timeline channels instead of forwarding raw message history.

agent-memory (context_management)

The Director's Journal acts as structured working memory with entry types, priority, follow-ups, citations, phase, round, and timestamp context.

eval-driven-development (quality)

The Critic Review and Timeline provide scored, evidence-based quality controls that prune low-credibility findings before they guide the investigation.

tool-use-orchestration (tooling)

The Critic uses explicit tools to inspect cited tool calls, results, toolset information, and available Expert toolsets instead of trusting summaries alone.

"We provide the Critic with four tools:

Tool

Purpose

get_tool_call

Inspect the arguments and metadata of any tool call"

Tradeoffs

Use curated context channels instead of complete history

Chose Journal, Review, and Timeline as online context summarization over Passing all message history between agent invocations.

Rationale: Curated context preserves continuity while avoiding context-window exhaustion, latency growth, cost growth, and information overload.

Use a stronger model for the Critic

Chose Higher-capability validation model for the bounded Critic task over Uniformly cheap or uniform-capability models across all roles.

Rationale: The Critic's scope is limited enough to keep token cost manageable while improving nuance interpretation and reducing errors.

Let the Critic inspect evidence through tools

Chose Summary report plus targeted review tools over Putting all raw evidence directly into the Critic prompt.

Rationale: Tool inspection scales better when many findings exist and lets the Critic inspect both evidence and methodology.

Gotchas (what bit them)

Naively accumulating message history breaks long-running agent workflows.

Root cause: Stateless LLM APIs require callers to provide continuity, but full history fills context windows and degrades quality.

Resolution: Slack replaced message-history carryover with Journal, Review, and Timeline context channels.

Sharing either too little or too much context can degrade multi-agent investigations.

Root cause: Agents need alignment to the broader investigation, but excessive context can stifle creativity and reinforce confirmation bias.

Resolution: Slack gives each role a tailored view of state through complementary context channels.

Expert findings can hallucinate or misinterpret evidence even under strict guidelines.

Root cause: Specialist agents produce findings from tools, but model outputs can invent or grossly misread supporting data.

Resolution: The Critic examines cited evidence and methodology, scores findings, and filters weak ones before timeline inclusion.

A model-based Critic can create false assurance if treated as infallible.

Root cause: The Critic's Review is itself model inference and can hallucinate or miss nuance.

Resolution: Slack mitigates this by using a stronger Critic model, narrow instructions, and an additional Timeline task.

The first mitigation is to use a stronger model for the Critic."

— article, stated

Evaluation

Slack evaluates investigation reliability through Review credibility scores, Timeline confidence scores, scoring distributions over reviewed findings, and specimen investigation traces showing whether Experts, Review, and Timeline converge on supported conclusions.

Metrics: 170,000 reviewed findings in the scoring distribution · 37.7% Trustworthy findings · 25.4% Highly-plausible findings · 11.1% Plausible findings · 10.4% Speculative findings · 15.4% Misguided findings · slightly over a quarter of findings below plausibility threshold · 0.83 confidence score in the example timeline

Outcomes

• The architecture maintains coherence across rounds without forwarding full message history.
• The Critic filters or down-ranks a large share of weak findings before they enter the consolidated timeline.
• The example investigation reached a high-confidence false-positive conclusion with all four Experts agreeing.
• The approach lets specialized agents retain role-specific context while preserving shared alignment.

Slack reports complex security investigations spanning hundreds of inference requests and megabytes of output, with Review scoring measured over 170,000 findings.