Building a Generic Recommender System API with Privacy-First Design

slack · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal recommender platform APIs

Slack needed a reusable recommender API for many small recommendation use cases across users and channels, while preserving strict customer privacy and avoiding repeated bespoke recommender implementations.

Constraints:

recommendations recur across many product contexts rather than one tent-pole use case
internal product teams need a simple API instead of ML-specific details
customer communications are highly sensitive and cannot leak across teams or workspaces
new recommenders have a cold-start problem before logged training data exists
feature engineering must be reusable but still adaptable to each use case

Architecture

Slack built a generic recommender API that accepts context about users and channels and returns recommended users or channels. Behind that simple API, the platform reuses candidate sources, online feature computation, logged features, a signal/feature service, hand-tuned cold-start models, and later ML models trained from accumulated data. Privacy constraints shape feature design, keeping sensitive message content out of features and preventing cross-customer leakage.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  internal-product-teams[Internal product teams] --> generic-recommender-api[Generic recommender API]
  generic-recommender-api --> candidate-generators[Candidate generators]
  candidate-generators --> feature-service[Feature service]
  feature-service --> feature-log[Feature log]
  feature-log --> model-training-pipeline[Model training pipeline]
  feature-service --> hand-tuned-ranking-model[Hand-tuned ranking model]
  hand-tuned-ranking-model --> ranking-models[Ranking models]
  model-training-pipeline --> ranking-models
  ranking-models --> generic-recommender-api
  privacy-constraints[Privacy constraints] --> feature-service
  privacy-constraints --> model-training-pipeline

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Internal product teams`	Consumers that need user or channel recommendations without owning ML infrastructure.	`Generic recommender API`
`Generic recommender API`	Simple reusable API boundary for recommendation queries and responses.	`Internal product teams`, `Candidate generators`, `Ranking models`
`Candidate generators`	Reusable candidate-fetching mechanisms such as memberships, operational stores, and embedding stores.	`Generic recommender API`, `Feature service`
`Feature service`	Computes reusable user and channel features for recommendation ranking.	`Candidate generators`, `Feature log`, `Ranking models`
`Feature log`	Stores features computed online so later models can train from production data.	`Feature service`, `Model training pipeline`
`Hand-tuned ranking model`	Explainable initial model used before enough training data exists.	`Feature service`, `Ranking models`
`Model training pipeline`	Trains learned recommendation models once logged data accumulates.	`Feature log`, `Ranking models`
`Ranking models`	Hand-tuned and learned models that score candidates for API responses.	`Generic recommender API`, `Feature service`, `Hand-tuned ranking model`, `Model training pipeline`
`Privacy constraints`	Design rules preventing message-content leakage and cross-customer data exposure.	`Feature service`, `Model training pipeline`

Data flow

An internal product calls the generic recommender API with a query/context. Candidate generation fetches possible users or channels from reusable sources such as memberships, operational stores, or embedding-neighbor stores. Feature services compute reusable user/channel features online, those features are logged for future training, and a hand-tuned model ranks early results until enough data exists for learned models.

3 supporting quotes

“we started off with sort of having an API where other teams could start calling these recommenders in a generic way”transcript, stated

“then we have a bunch of features like everyone actually built out this signal service that we can use so we have all the features about users and channels”transcript, stated

“you log what features are computed and then you can use those later for training so that means that you need to start with something that'll make a reasonably good prediction so that before you have any training data okay could you like and that's and that's”transcript, stated

Why this architecture

Slack hid recommender complexity behind a strong, simple API so internal teams could integrate recommendations without understanding embeddings, ranking, or feature pipelines. The platform starts with explainable hand-tuned models for cold start, logs online features for training, and applies privacy constraints from the ground up because leaked Slack data would destroy more value than recommendations create.

3 supporting quotes

“people off but if you just say hey here's an API here's a basic idea of what you said this is query we'll we'll do the rest and send you back good recommendations I think that kind of that made it very easy to start onboarding other teams and that allowed us to sort of start building back from”transcript, stated

“pretty intuitive right we can always start with a handshake model and then why is that we can gather Trinity data the other is that like people are just like when we explain that the model like the hand to the model to PMS they are like how often it for example we are”transcript, stated

“it's hard to imagine how the value we can create with ML would outweigh the value we could destroy if we leak customers data”transcript, stated

Models used: hand-tuned ranking models · ML recommendation models

Scale

Slack described around a hundred recommendation-style product contexts.
“some context like there's a there's a hundred different products you can kind of set up that followed basically that framework so we kind of started thinking well how can we make this generic both in terms of like how we present it to other teams at the company and how we”transcript, stated
The platform had hundreds of reusable user/channel features.
“um and are using for for ranking there's probably tens hundreds thousands hundreds I mean I would say there's hundreds um but in practice you know I think it's usually most models maybe need a couple dozen and we could probably get down to just a”transcript, stated
Most models need a couple dozen features rather than the full feature inventory.
“most models maybe need a couple dozen and we could probably get down to just a couple bills of across all our models”transcript, stated

Alternatives considered (roads not taken)

Bespoke recommender per product use case

Rejected because: Slack repeatedly saw similar recommendation problems and wanted one reusable API plus shared backend tools.

“he had sort of solved the same problem basically the same way a couple times he's like huh we should really make this generic”transcript, stated

Expose ML concepts directly to product teams

Rejected because: Talking about embeddings and models could throw teams off, while a simple API made onboarding easier.

“if you start throwing around talking about embeddings or machine learning models that can kind of throw people off”transcript, stated

Use customer message text as recommendation features

Rejected because: Message content creates leakage and privacy risks that outweigh ML value.

“anything where we're dealing with like customer text Data the things you write into slack we have to be like really really careful about them”transcript, stated

Wait for learned models before launching a recommender

Rejected because: Hand-tuned models provide an explainable first version and generate training data for later learned models.

“of mesh it implicitly I think we can start with the heroes models and then just use that to gather the training data I just never thought of this before I joined stack and then after I joined slack and see how people actually like how the team is actually doing this it's”transcript, stated

Patterns

`privacy-preserving-processing` (privacy)

Slack avoids message-content-derived features and designs models not to leak customer data or even imply private DM existence.

“we don't even want to like we want to design our features where we don't even hint at the existence of a DM between you and another person”transcript, stated

`structured-output-contract` (api-design)

The recommender API gives internal teams a simple query-in/recommendations-out contract while hiding model complexity.

“strong simple API that people just get the right thing from”transcript, stated

`data-quality-loop` (data)

The system logs online computed features so production usage can become training data for later models.

“you are doing all the feature engineering online uh your logging the features that are computed”transcript, stated

`eval-driven-development` (evaluation)

Hand-tuned models provide an understandable baseline that learned models can beat before being trusted.

“it gives you a reasonable Benchmark to beat what I saw from the slack stats was actually the email models did come in and and wipe the floor with a lot of these hand to models”transcript, stated

`retrieval-ranking` (ranking)

The platform separates candidate fetching from ranking so use cases can plug in different sources and feature weights.

“the first stage is always fetching the candidates and then maybe we are going to use different features”transcript, stated

Tradeoffs

Use a simple API boundary

Chose query/context in, recommended users/channels out over exposing the full feature/model stack to every product team.

Rationale: A simple API made onboarding easier while letting the ML team improve complexity behind it.

“made it very easy to start onboarding other teams and that allowed us to sort of start building back from that basic API”transcript, stated

Prioritize privacy over potentially useful text features

Chose interaction and public-channel features over message-content features.

Rationale: Recommendation lift was not worth the risk of exposing customer data.

“things we do which would leak that data are not even worth considerate”transcript, stated

Start with hand-tuned models

Chose explainable weights and feature rules over waiting for trained ML models.

Rationale: Hand-tuned models were explainable to PMs and generated data for later ML improvements.

“they just understand the hand tool model and then they can see the results also is already making sense”transcript, stated

Gotchas (what bit them)

A generic recommender can leak sensitive customer relationships.

Root cause: Slack communication data contains private content and relationships; model outputs can reveal them indirectly.

Resolution: Design features around public-channel interactions and avoid private message content or relationship leakage.

“we build a lot of features exclusively from public Channel messages or public Channel interactions to make sure we're not leaking that stuff”transcript, stated

Product teams can be blocked by ML vocabulary.

Root cause: Embeddings and model internals are not the right abstraction for internal API customers.

Resolution: Expose a simple API boundary and keep ML complexity behind the platform.

“how to make a strong simple easy to integrate with API that you can have an excellent ml engineering team behind”transcript, stated

Cold-start recommenders lack training data.

Root cause: Logged features and labels do not exist before the first version runs.

Resolution: Start with hand-tuned models, log online features, then train learned models once data accumulates.

“you need to start with something that'll make a reasonably good prediction so that before you have any training data”transcript, stated

Evaluation

Slack evaluated early recommenders by comparing learned models against hand-tuned baselines and by using logged online features for training and iteration.

Metrics: hundreds of reusable features · couple dozen features per typical model · hand-tuned baseline beaten by learned models · privacy review for leakage risk

“the email models did come in and and wipe the floor with a lot of these hand to models”transcript, stated

“we were testing out something internally and we scrapped it because we realized like this it could indicate that like two people were messaging each other”transcript, stated

Outcomes

Slack created a generic recommender API reusable across many internal user/channel recommendation contexts
internal teams could integrate recommendations without owning ML internals
hand-tuned models handled cold start and later learned models improved over them
privacy constraints shaped feature design and blocked risky experiments

The transcript gives approximate scope: roughly a hundred product contexts and hundreds of reusable features, but not a public production request volume.