Scaling AI-Assisted Developer Tools and Agentic Workflows at Scale

slack · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal developer productivity and escalation workflows

Slack needed to scale AI-assisted developer tools from experiments to production across its engineering organization, reducing friction in coding and escalations while meeting FedRAMP/security requirements, controlling cost, preserving observability, and supporting durable multi-agent workflows.

Constraints:

• Slack's DevEx team supports the full Slack engineering organization and broader Salesforce contexts.
• Early SageMaker experimentation gave control but carried infrastructure maintenance cost.
• FedRAMP compliance influenced model-platform choices.
• Slack needed to reduce experimentation fatigue from many competing LLM tools.
• The escalation bot needed durable conversation state, secure internal-tool access, parallel sub-agents, retries, and model agnosticism.

Architecture

Slack moved from SageMaker experimentation to Bedrock for compliant, managed model access, then rolled out Cursor and Claude Code through Bedrock. Buddy Bot evolved from a RAG/search assistant into an escalation agent using Temporal for durable conversation workflows, Strands as the orchestrator, Claude Code sub-agents for triage and knowledge tasks, MCP servers for internal service access, and Bedrock/CloudWatch/OpenTelemetry-style observability.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Slack developers"] --> B["Claude Code and Cursor"]
  C["Bedrock platform"] --> B
  A --> D["Escalation bot"]
  D --> E["Temporal workflow"]
  E --> F["Strands orchestrator"]
  C --> F
  F --> G["Claude Code sub-agents"]
  F --> H["MCP servers"]
  G --> H
  H --> I["Internal services"]
  D --> A

Components

Data flow

Developers use Bedrock-backed AI coding tools for code work, while escalation requests arrive in Slack channels. The backend receives an event, starts or resumes a Temporal workflow with the Slack-thread context, calls a Strands orchestrator, launches Claude Code sub-agents through tool calls, uses MCP servers to reach internal services, summarizes sub-agent outputs to manage token use, and sends a response back to the Slack channel.

Why this architecture

Slack standardized around Bedrock and Anthropic tooling to reduce infrastructure overhead, maintain FedRAMP/security posture, and curb tool fatigue. For the escalation bot, Temporal removed the burden of conversation history and retry durability, while Strands preserved model agnosticism and MCP enabled secure tool access.

Models used: Anthropic Claude · Claude Code · Cursor · Bedrock models · Strands agents

Scale

• Slack reports 99% of developers using some kind of AI assistance.
  “Currently 99% of our developers are using some kind of AI assistance, which is huge.”transcript, stated
• Major repositories saw about 25% month-over-month PR throughput increase.
  “we are seeing about 25% consistent month over month increase in the PR throughput.”transcript, stated
• The AI assistance bot helps with more than 5,000 escalation requests per month.
  “currently our AI assistance Bot is helping over 5,000 escalation requests per month.”transcript, stated
• Moving to Bedrock saved nearly 98% of costs.
  “they nearly saved 98% of the costs when they moved to Bedrock.”transcript, stated
• Slack scaled from a few hundred thousand tokens per minute to millions of tokens per minute.
  “they have scaled from processing a few 100,000 tokens per minute to millions of tokens per minute.”transcript, stated

Alternatives considered (roads not taken)

Continue SageMaker-managed experimentation

Rejected because: SageMaker gave control but required infrastructure maintenance; Bedrock simplified LLM scale and maintenance.

Keep rolling out many competing AI tools

Rejected because: Too many internal features created confusion and maintenance overhead, so Slack standardized on a high-impact stack.

Use only Claude Code sub-agents without Strands orchestration

Rejected because: Slack wanted to explore other models and preserve extensibility instead of sticking only with Claude Code.

Patterns

multi-agent-orchestration (orchestration)

Slack uses a Strands orchestrator to select Claude Code sub-agents such as triage and knowledge-base agents, then synthesizes their outputs.

durable-execution (orchestration)

Temporal provides durable conversation workflows, retries, state storage, and resumption after failures.

tool-use-orchestration (orchestration)

MCP servers give the bot controlled access to internal services such as GitHub and other sensitive tools.

model-routing (model-ops)

Slack chose Strands and Bedrock to remain model agnostic while experimenting beyond Claude Code.

observability-tracing (observability)

Slack uses Bedrock/CloudWatch observability and Temporal traceability to understand LLM usage, calls, and workflow behavior.

Tradeoffs

Managed model platform

Chose Amazon Bedrock over continuing SageMaker/RAG infrastructure ownership.

Rationale: Bedrock met FedRAMP needs, simplified infrastructure, gave access to Anthropic models, and sharply reduced cost.

Agent orchestration

Chose Strands orchestrator with Claude Code sub-agents over Claude Code sub-agents alone.

Rationale: Strands lets Slack explore other models and keep the orchestration layer extensible.

State management

Chose Temporal workflow state over application-owned conversation history and retry code.

Rationale: Temporal simplified Slack's code by handling conversational history, state, retries, and resume behavior.

Gotchas (what bit them)

Infrastructure control came with hidden maintenance cost.

Root cause: The SageMaker/RAG approach required Slack to manage more infrastructure.

Resolution: Slack moved to Bedrock once it was FedRAMP compliant and could handle LLM scale and maintenance.

Rapid tool proliferation caused developer confusion.

Root cause: The AI landscape changed quickly and rolling out competing internal features created maintenance overhead.

Resolution: Slack doubled down on Bedrock, Anthropic models, Claude Code, and Cursor as a high-impact stack.

More AI-generated code increased review load.

Root cause: AI helped engineers write more code, increasing the surface area for peer review.

Resolution: Slack is working on AI assistance to reduce review time across the developer cycle.

Escalation conversations require durable state and retries.

Root cause: Slack thread escalations continue over time and the backend can fail while a conversation is unresolved.

Resolution: Temporal stores state, resumes workflows, and provides automated retries.

Evaluation

Slack measures adoption, usage, PR throughput, escalation volume, developer feedback, review-time side effects, Bedrock observability, CloudWatch logs/metrics/alerts, and Temporal traceability.

Metrics: 99% developer adoption · 25% month-over-month PR throughput increase · 5,000+ escalation requests per month · 98% cost reduction moving to Bedrock · Tokens per minute · Peer review time · Developer feedback

Outcomes

• Slack moved from SageMaker experimentation to Bedrock.
• Moving to Bedrock reportedly saved nearly 98% of costs.
• AI assistants reached 99% developer adoption.
• Major repositories saw about 25% month-over-month PR throughput increase.
• The AI assistance bot handles over 5,000 escalation requests per month.
• Buddy Bot evolved into a durable Strands/Temporal/MCP multi-agent escalation bot.

Slack reports 99% developer adoption, 25% PR throughput increase in major repositories, over 5,000 monthly escalation requests, nearly 98% cost savings after Bedrock migration, and token processing growth from hundreds of thousands per minute to millions per minute.