Building Secure and Private Enterprise LLM Infrastructure

slack · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise collaboration AI infrastructure

Slack wanted to add LLM-powered summarization and search while preserving enterprise privacy, compliance, access-control, and data-retention guarantees for customer workspace data.

Constraints:

Architecture

Slack AI uses an enterprise trust-boundary architecture. Rather than call hosted model APIs directly, Slack uses AWS as a trusted broker and hosts closed-source LLMs in an escrow VPC inside Slack's trust boundary, so model providers cannot access customer data. Slack avoids customer-data training by using stateless RAG: each request includes the messages or search context needed for the task, and the model retains none of it. Data retrieval reuses existing Slack ACL-aware channel/search libraries so Slack AI sees only what the requesting user can see. Outputs are visible only to the invoker, ephemeral where possible, and derived summaries remain linked to source messages so DLP tombstones and administrative controls invalidate AI outputs too.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  requesting-user[Requesting user] --> slack-ai-feature[Slack AI feature]
  slack-ai-feature --> acl-aware-data-fetcher[ACL aware data fetcher]
  acl-aware-data-fetcher --> workspace-message-store[Workspace message store]
  acl-aware-data-fetcher --> rag-prompt-builder[RAG prompt builder]
  rag-prompt-builder --> escrow-vpc-llm-endpoint[Escrow VPC LLM endpoint]
  aws-trusted-broker[AWS trusted broker] --> escrow-vpc-llm-endpoint
  escrow-vpc-llm-endpoint --> ai-output-renderer[AI output renderer]
  slack-ai-feature --> ai-output-renderer
  ai-output-renderer --> derived-content-tracker[Derived content tracker]
  workspace-message-store --> compliance-control-layer[Compliance control layer]
  derived-content-tracker --> compliance-control-layer
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Requesting user Slack user invoking AI summarization or search. Slack AI feature
Slack AI feature Product layer for channel summaries and AI search answers. Requesting user, ACL aware data fetcher, RAG prompt builder, AI output renderer
ACL aware data fetcher Existing Slack channel/search data-access libraries that enforce the requesting user's visibility rules. Slack AI feature, Workspace message store, RAG prompt builder
Workspace message store Customer workspace data, messages, and search content governed by Slack's existing controls. ACL aware data fetcher, Compliance control layer
RAG prompt builder Assembles task instructions plus all context needed for the model in each stateless request. ACL aware data fetcher, Escrow VPC LLM endpoint
AWS trusted broker Broker arrangement that lets Slack host closed-source LLMs without exposing customer data to model providers. Escrow VPC LLM endpoint
Escrow VPC LLM endpoint Closed-source LLM hosted within Slack's trust boundary for summarization and search generation. AWS trusted broker, RAG prompt builder, AI output renderer
AI output renderer Displays generated summaries/search answers only to the user who invoked Slack AI. Slack AI feature, Escrow VPC LLM endpoint, Derived content tracker
Derived content tracker Tracks relationships between AI-generated summaries and source messages when outputs must persist. AI output renderer, Compliance control layer
Compliance control layer Slack enterprise controls including DLP, EKM, International Data Residency, and data-retention behavior. Workspace message store, Derived content tracker

Data flow

A requesting user invokes Slack AI for search or channel summarization. Slack fetches only the messages or search results that user can already see by reusing existing ACL-aware channel/search libraries. The RAG prompt builder packages the task instructions and retrieved context into a stateless request to a closed-source LLM hosted in an AWS escrow VPC within Slack's trust boundary. The output is shown only to the invoking user and, where possible, is ephemeral rather than stored on disk. When derived content must persist, Slack tracks which source messages went into it so DLP tombstones and administrative controls invalidate the derived output.

5 supporting quotes
“We ensure this by using the requesting user’s Access Control List (ACLs) when fetching the data to summarize or search”article, stated
“when summarizing a channel, we’ll send the LLM a prompt containing the messages to be summarized, along with instructions for how to do so.”article, stated
“By using AWS, we are able to host and deploy closed-source large language models (LLMs) in an escrow VPC, allowing us to control the lifecycle of our customers’ data and ensure the model provider has no access to Slack’s customers’ data. We had access to a”article, stated
“Slack’s core feature sets while adding some AI magic at the end. It’s worth noting, too, that only the user who invokes Slack AI can see the AI-generated output. This builds confidence that Slack is your trusted AI partner: Only the data that you can see goes in, and then only you”article, stated
“if a message is tombstoned because of Data Loss Protection (DLP), any summaries derived from that message are invalidated.”article, stated

Why this architecture

Slack chose this architecture because direct hosted-model calls and customer-data fine-tuning conflicted with its enterprise trust model. The escrow VPC keeps customer data inside Slack's boundary while still allowing top-tier closed-source models. Stateless RAG avoids model retention/training and grounds outputs in workspace knowledge. Reusing core ACL and compliance systems prevents AI from becoming a parallel data-access path with weaker governance.

4 supporting quotes
“We knew this approach wouldn’t work for us. Slack, and our customers, have high expectations around data ownership.”article, stated
“we couldn’t make strong enough guarantees on these practices if we trained a generative AI model using Slack’s customers’ data.”article, stated
“The statelessness of RAG is a huge privacy benefit, but it’s a product benefit as well. All of Slack AI’s results are grounded in your company’s knowledge base — not the public Internet – which makes the results more relevant and accurate. You get the”article, stated
“the best way to guarantee this was to build on top of, and reuse, Slack’s core feature sets while adding some AI magic at the end.”article, stated

Models used: Closed-source large language models hosted in an AWS escrow VPC · Traditional Slack ML models chained with generative models

Scale

Alternatives considered (roads not taken)

Call hosted foundational model services directly

Rejected because: Direct API calls would send customer data outside Slack's trust boundary and risk third-party retention/training.

“most customers of foundational models were calling the hosted services directly, and alternative options were scarce.”article, stated
“foundational models were calling the hosted services directly, and alternative options were scarce. We knew this approach wouldn’t work for us. Slack, and our customers, have high expectations around data ownership. In particular, Slack”article, stated

Self-host closed-source foundational models directly on Slack hardware

Rejected because: Most top foundational models are closed-source and providers do not hand them over for customer deployment.

“most foundational models are closed-source: Their models are their secret sauce, and they don’t like to hand them to customers to deploy on their own hardware.”article, stated

Train or fine-tune LLMs on customer data

Rejected because: Slack could not guarantee workspace isolation and customer choice strongly enough in the young generative-AI ecosystem.

“(LLMs) on customer data The next decision was also key: We chose to use off-the-shelf models instead of training or fine-tuning models. We’ve had privacy principles in place since we began employing more traditional machine learning (ML)”article, stated
“we couldn’t make strong enough guarantees on these practices if we trained a generative AI model using Slack’s customers’ data.”article, stated

Build separate AI-specific data access paths

Rejected because: AI should inherit standard Slack search/channel visibility rules rather than become a new bypassable path.

“leveraging our existing libraries that fetch the data to display in channel or on the search results page. This wasn’t hard to do, technically speaking, but it needed to be an explicit choice; the best way to guarantee this was to build on top of, and reuse,”article, stated
“in channel or on the search results page. This wasn’t hard to do, technically speaking, but it needed to be an explicit choice; the best way to guarantee this was to build on top of, and reuse, Slack’s core feature sets while adding some AI magic”article, stated

Persist all AI outputs like ordinary content

Rejected because: Slack follows least-data principles and makes summaries/search answers ephemeral where possible.

“We follow the principle of least data: We store only the data needed to complete the task, and only for the duration necessary.”article, stated
“Where possible, Slack AI’s outputs are ephemeral: Conversation summaries and search answers all generate point-in-time responses that are not stored on disk.”article, stated

Patterns

privacy-preserving-processing (security)

Slack keeps customer data inside its trust boundary by running closed-source LLMs through an AWS escrow VPC and using stateless RAG instead of customer-data training.

“* Customer data never leaves Slack’s trust boundary. * We do not train large language models (LLMs) on customer data.”article, stated
“allowing us to control the lifecycle of our customers’ data and ensure the model provider has no access to Slack’s customers’ data.”article, stated

rag-retrieval (retrieval)

Slack sends task-specific context in each prompt so AI answers are grounded in workspace knowledge without model retention.

“we made the choice to use off-the-shelf models in a stateless way by employing Retrieval Augmented Generation (RAG).”article, stated
“All of Slack AI’s results are grounded in your company’s knowledge base — not the public Internet”article, stated

safety-policy-enforcement (security)

Slack AI reuses user ACLs and existing Slack feature libraries so summaries/search cannot expose content the user could not already access.

“Slack AI can only see the same data that the requesting user can see.”article, stated
“Slack AI’s search feature, for example, will never surface any results to the user that standard search would not.”article, stated

guardrails-io-filtering (compliance)

Derived Slack AI outputs remain subject to DLP and administrative controls, including invalidation when source messages are tombstoned.

“if a message is tombstoned because of Data Loss Protection (DLP), any summaries derived from that message are invalidated.”article, stated
“Where these controls were already active on Slack’s message content, they are also active Slack AI outputs.”article, stated

latency-optimization (performance)

Slack's model selection had to balance large context windows with low latency for summarization and search.

“the more context you send an LLM, the slower your request will be”article, stated
“Find a top-tier model with a large context window with fairly low latency.”article, stated

model-routing (architecture)

Slack evaluated multiple commercial models and selected one suited to its initial summarization and search use cases.

“We evaluated a number of models and found one that suited our first use cases, summarization and search, well.”article, stated

Tradeoffs

Host commercial LLMs through AWS escrow VPC

Chose Top-tier closed-source model inside Slack's trust boundary over Direct hosted model API calls or unavailable self-hosted closed models.

Rationale: Slack wanted leading model capability without allowing customer data to leave its controlled boundary.

“ensure that we could use a top-tier foundational model while never allowing customer data to leave Slack’s trust boundary.”article, stated
“AWS has an offering where it can be the trusted broker between foundational model provider and customer.”article, stated

Use stateless RAG instead of training

Chose Per-request retrieved context over Training or fine-tuning LLMs on customer data.

Rationale: RAG provides privacy and product relevance while avoiding model retention of customer data.

“With RAG, you include all of the context needed to perform a task within each request, so the model does not retain any of that data.”article, stated
“You get the benefit of incorporating your proprietary and individual data set without the risk of a model retaining that data.”article, stated

Reuse existing Slack ACL and compliance systems

Chose Build on core search/channel display libraries over Parallel AI-specific access and compliance infrastructure.

Rationale: Existing systems already encoded user visibility and enterprise controls, making them the safest base for AI features.

“leveraging our existing libraries that fetch the data to display in channel or on the search results page.”article, stated
“reused as much of Slack’s existing compliance infrastructure as possible, and built new support where we had to.”article, stated

Keep outputs ephemeral where possible

Chose Point-in-time non-disk summaries and search answers over Persisting all AI-generated outputs.

Rationale: Least-data storage reduces exposure and compliance burden for generated content.

“Sometimes the least data is: None. Where possible, Slack AI’s outputs are ephemeral: Conversation summaries and search answers all generate point-in-time responses that are not stored on disk.”article, stated
“Conversation summaries and search answers all generate point-in-time responses that are not stored on disk.”article, stated

Accept RAG latency and context-window constraints

Chose Large-context, low-latency model selection and prompt tuning over Quality gains from customer-data fine-tuning.

Rationale: RAG better preserved customer-data guarantees even though it constrained model choice and introduced latency pressure.

“Using RAG can narrow down the set of models you can use; they need to have “context windows” large enough for you to pass in all the data you want to use in your task.”article, stated
“There was room for improvement, though, and we began a long journey of both prompt tuning and chaining more traditional ML models with the generative models”article, stated

Gotchas (what bit them)

Direct hosted LLM APIs can violate enterprise trust-boundary requirements.

Root cause: Customer data may leave the controlled environment and become available for retention or provider training.

Resolution: Use an AWS escrow VPC arrangement so Slack controls customer data lifecycle while accessing closed-source models.

“including not sending customer data outside of our trust boundary.”article, stated
“By using AWS, we are able to host and deploy closed-source large language models (LLMs) in an escrow VPC”article, stated

Customer-data fine-tuning can undermine workspace isolation and choice guarantees.

Root cause: The generative AI ecosystem was too young for Slack to make strong enough guarantees about training on customer data.

Resolution: Use off-the-shelf models in a stateless RAG pattern instead of training or fine-tuning on customer data.

“* We do not train large language models (LLMs) on customer data. * Slack AI only operates on the data that the user can already see.”article, stated
“we couldn’t make strong enough guarantees on these practices if we trained a generative AI model using Slack’s customers’ data.”article, stated

AI features can accidentally bypass existing authorization if they fetch data through new paths.

Root cause: A separate AI data layer might not inherit normal search/channel ACL behavior.

Resolution: Build Slack AI on existing ACL-aware libraries and core feature sets.

“We ensure this by using the requesting user’s Access Control List (ACLs) when fetching the data to summarize or search”article, stated
“the best way to guarantee this was to build on top of, and reuse, Slack’s core feature sets”article, stated

RAG can make model selection and latency harder.

Root cause: The whole task context must fit in the model context window, and larger context slows requests.

Resolution: Evaluate models for large context windows and acceptable latency, then tune prompts and chain traditional ML with generative models.

“Using RAG can narrow down the set of models you can use; they need to have “context windows” large enough”article, stated
“Additionally, the more context you send an LLM, the slower your request will be”article, stated

Derived AI content can become stale or noncompliant when source messages change.

Root cause: Summaries depend on source messages but are separate generated artifacts.

Resolution: Track relationships between summaries and source messages so DLP tombstones invalidate derived summaries.

“built in special support to make sure that derived content, like summaries, are aware of the messages that went into them”article, stated
“if a message is tombstoned because of Data Loss Protection (DLP), any summaries derived from that message are invalidated.”article, stated

Evaluation

The article says Slack evaluated a number of models for its first summarization and search use cases and engaged in prompt tuning, but it does not disclose detailed evaluation methodology.

Metrics: Large enough context window · Fairly low latency · Suitability for summarization · Suitability for search · User-reported productivity

4 supporting quotes
“Find a top-tier model with a large context window with fairly low latency.”article, stated
“We evaluated a number of models and found one that suited our first use cases, summarization and search, well.”article, stated
“I didn’t even get to take you through how we build prompts, evaluate models, or handle spiky demand; we’ll save that for next time.”article, stated
“90% of users who adopted AI reported a higher level of productivity than those who didn’t.”article, stated

Outcomes

Slack AI is designed for enterprise Slack customers under existing compliance offerings, including FedRAMP Moderate, EKM, International Data Residency, and DLP.

5 supporting quotes
“We had access to a top tier foundational model, hosted within our trust boundary, giving us assurances on our customer data.”article, stated
“ensure the model provider has no access to Slack’s customers’ data.”article, stated
“Summaries will never summarize content that the user could not otherwise see while reading channels.”article, stated
“Where these controls were already active on Slack’s message content, they are also active Slack AI outputs.”article, stated
“90% of users who adopted AI reported a higher level of productivity than those who didn’t.”article, stated