Scaling Generative AI Features to Millions of Users with Infrastructure Optimization and Quality Evaluation

slack · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise collaboration GenAI infrastructure

Slack needed to scale Slack AI to millions of users while preserving customer trust, compliance, quality, model flexibility, and cost efficiency.

Constraints:

• Slack AI needed to serve requests within Slack's trust boundary and satisfy FedRAMP Moderate requirements.
• SageMaker-era provisioned infrastructure was less flexible and more costly as usage scaled.
• Quality had to be measured across complex generated outputs, not just infrastructure metrics.

Architecture

Slack migrated from SageMaker endpoints to Amazon Bedrock, initially matching provisioned-throughput capacity, using load tests to map compute units, shadow traffic to verify latency and metrics, staged cutover percentages, model fallback/switching, and an internal experimentation framework with automated metrics, LLM judges, and guardrails.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Slack AI features"] --> B["Migration router"]
  B --> C["SageMaker endpoints"]
  B --> D["Amazon Bedrock"]
  D --> E["Model registry"]
  D --> F["Fallback model layer"]
  A --> G["Experimentation framework"]
  F --> G
  G --> H["LLM judges and metrics"]

Components

Data flow

Existing SageMaker traffic was duplicated to Bedrock as shadow traffic, Slack monitored latencies and time-to-first-token, then cut over from 1% to 100%. Bedrock's model registry and trust boundary enabled production model experimentation across more than 15 LLMs. Offline and online experiments combine automated programmatic metrics, LLM judges, guardrails, and user feedback to evaluate quality.

Why this architecture

Bedrock gave Slack managed serving, model diversity, trust-boundary controls, and faster access to frontier models, while staged migration and experimentation protected reliability and quality during scale-up.

Models used: Claude Instant · Claude Haiku · Amazon Bedrock LLMs · more than 15 LLMs in production · LLM judges

Scale

• Slack experimented with and served more than 15 LLMs in production.
  “we're experimenting and serving greater than 15 LLMs in production.”transcript, stated
• The migration produced greater than 90% savings, exceeding $20 million.
  “we saw greater than 90% savings and in dollar values, this was greater than 20 million, as well.”transcript, stated
• Slack AI cost to serve per monthly active user fell 90%.
  “we've seen a 90% reduction in the cost to serve Slack AI by monthly active user”transcript, stated
• Slack increased operating scale by nearly 5x.
  “we've increased the scale that we operate at by nearly 5X.”transcript, stated
• User satisfaction and feedback improved by 15-30% across marquee features.
  “increasing user satisfaction and user feedback by ranging from 15 to 30% across Slack AI's marquee features.”transcript, stated

Alternatives considered (roads not taken)

Stay on SageMaker provisioned infrastructure

Rejected because: Slack wanted managed serving, model diversity, faster model availability, and better cost flexibility.

Immediate full cutover

Rejected because: Slack used shadow traffic and staged 1/5/10/100% cutover to preserve trust and avoid downtime.

Manual quality review only

Rejected because: Slack needed large-scale representative evaluation over production-like datasets.

Patterns

canary-rollout (deployment)

Slack shadowed SageMaker traffic to Bedrock, then cut over through 1%, 5%, 10%, and 100% stages.

model-routing (model-ops)

Slack serves and experiments with more than 15 LLMs and can switch or fall back during incidents.

llm-as-judge (evaluation)

Slack uses LLM-based quality metrics and judges to evaluate factual accuracy, answer relevance, and other quality criteria.

guardrails-io-filtering (safety)

Slack includes guardrails in automated quality evaluation to capture safety and harm issues.

Tradeoffs

Migration path

Chose shadow traffic and staged cutover over direct SageMaker-to-Bedrock cutover.

Rationale: This preserved customer trust, enabled dashboards, and verified latency/time-to-first-token before serving Bedrock responses.

Initial Bedrock capacity mode

Chose provisioned throughput parity first over immediate on-demand transition.

Rationale: Staying close to SageMaker provisioned infrastructure made migration easier before optimizing on-demand later.

Gotchas (what bit them)

Migration needed compute equivalence, not guesswork.

Root cause: SageMaker and Bedrock capacity units did not map intuitively across model and instance types.

Resolution: Run load tests and calculate equivalent Bedrock model units before migration.

Bedrock model regressions were possible.

Root cause: Model availability and quality changed as Slack expanded beyond the old platform.

Resolution: Use experimentation, monitoring, and fallback/switching ability across models.

Quality is not captured by one metric.

Root cause: Slack AI features have complex definitions of quality such as factual accuracy, relevance, safety, and user feedback.

Resolution: Combine programmatic metrics, LLM judges, guardrails, and feature-specific quality evaluation.

Evaluation

Slack evaluates infrastructure and quality using load tests, shadow traffic, latency/time-to-first-token metrics, cost-to-serve, scale, user satisfaction, LLM judges, guardrails, and offline/online experiments.

Metrics: latency · time to first token · cost savings · cost to serve per MAU · scale multiplier · user satisfaction · factual accuracy · answer relevance · safety/harm

Outcomes

• Slack reported greater than 90% savings worth more than $20 million during migration.
• Cost to serve Slack AI per monthly active user fell 90%.
• Operating scale increased nearly 5x.
• User satisfaction and feedback improved by 15-30% across marquee features.
• Slack served and experimented with more than 15 LLMs in production.

The case emphasizes infrastructure migration and experimentation framework maturity as much as product metrics.