Multi-Cloud LLM Infrastructure Evolution at Scale

slack · Tech · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM infrastructure

Slack needed to evolve LLM serving infrastructure for enterprise AI features across security, compliance, capacity, latency, model freshness, and provider-risk requirements.

Constraints:

• Slack serves enterprise users with sensitive customer data.
• Early SageMaker deployments required substantial operational plumbing.
• GPU scarcity and container startup latency made capacity planning hard.
• Single-provider dependence limited model freshness and workload optimization.

Architecture

Slack evolved through four phases: SageMaker in an escrow VPC, migration to Bedrock, hybrid provisioned/on-demand capacity, and a multi-cloud architecture using AWS and Google Vertex AI for dynamic workload orchestration and best-fit model selection.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-ai-features["Slack AI Features"]
  multi-cloud-orchestration-layer["Multi-Cloud Orchestration Layer"]
  sagemaker-foundation["SageMaker Foundation"]
  amazon-bedrock["Amazon Bedrock"]
  google-vertex-ai["Google Vertex AI"]
  slack-ai-features --> multi-cloud-orchestration-layer
  multi-cloud-orchestration-layer --> amazon-bedrock
  multi-cloud-orchestration-layer --> google-vertex-ai
  multi-cloud-orchestration-layer --> sagemaker-foundation
  sagemaker-foundation --> multi-cloud-orchestration-layer
  amazon-bedrock --> multi-cloud-orchestration-layer
  google-vertex-ai --> multi-cloud-orchestration-layer

Components

Data flow

Slack AI features route LLM workloads through a multi-cloud orchestration layer that can select managed model providers and capacity modes based on feature needs, latency, quality, cost, compliance, and availability.

Why this architecture

Slack moved to multi-cloud because managed LLM infrastructure reduced operational burden, improved model freshness, mitigated provider concentration risk, and let different AI features use different models and capacity modes.

Models used: Anthropic models · Bedrock-hosted LLMs · Vertex AI-hosted LLMs

Scale

• The infrastructure serves millions of enterprise users.
  “serving millions of users across their AI-powered features including channel summaries, AI search, and daily recaps.”summary, stated
• The evolution spanned four phases across 2023-2026.
  “through four distinct phases over three years (2023-2026) to serve AI features to millions of enterprise users.”summary, stated
• The multi-cloud approach produced about 10% quality gain and about 67% latency reduction for selected workloads.
  “delivers measurable improvements including ~10% quality gains for reasoning tasks and ~67% latency reduction for high-velocity workloads”summary, stated

Alternatives considered (roads not taken)

Stay on SageMaker-only managed serving

Rejected because: SageMaker required custom scaling, capacity planning, and lagged newer model releases.

Use a single managed LLM provider

Rejected because: Single-cloud dependence created provider risk and constrained feature-specific model choice.

Patterns

model-routing (architecture)

Slack routes workloads across Bedrock and Vertex AI to match model, cost, latency, and quality needs.

safety-policy-enforcement (governance)

Security, FedRamp, and escrow VPC constraints shaped initial LLM serving architecture.

observability-tracing (operations)

The case emphasizes migration discipline and incident-free reliability across infrastructure changes.

Tradeoffs

Migrate from SageMaker to Bedrock

Chose Fully managed LLM service with faster model access over Custom-managed SageMaker serving.

Rationale: The team wanted operational simplicity, model freshness, and capacity flexibility.

Adopt multi-cloud

Chose AWS plus Google Vertex AI routing over Single-provider managed LLM infrastructure.

Rationale: Multi-cloud reduced single-provider risk and improved workload-specific model choice.

Gotchas (what bit them)

Managed ML containers did not scale instantly.

Root cause: SageMaker container initialization made traffic-spike response slow.

Resolution: Slack used proactive capacity planning and eventually migrated to Bedrock and multi-cloud capacity models.

GPU scarcity forced idle capacity costs.

Root cause: Enterprise-grade Nvidia GPU instances were often unavailable when needed.

Resolution: Slack used reservations and proactive scaling before later shifting toward managed LLM capacity.

Evaluation

Slack evaluated infrastructure through quality, latency, migration reliability, compliance, and incident metrics.

Metrics: ~10% quality gain · ~67% latency reduction · zero customer-facing incidents

Outcomes

• Slack moved from SageMaker to Bedrock and then to multi-cloud LLM routing.
• The infrastructure supports AI search, channel summaries, and daily recaps for millions of users.
• The architecture delivered reported quality and latency improvements without customer-facing migration incidents.

Millions of enterprise users, four infrastructure phases across 2023-2026, and reported ~10% quality and ~67% latency improvements.