Context-Driven AI Data Assistant for Enterprise Data Warehousing

spotify · Media & Entertainment · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise data access and SQL analytics

Spotify needed a way for thousands of teams to ask data questions across a huge warehouse without waiting on individual domain experts or giving an LLM raw schemas it could misunderstand.

Constraints:

Spotify's warehouse spans more than 70,000 datasets and petabytes of data.
Raw schemas do not encode business definitions, legacy quirks, or canonical query patterns.
Answers had to be transparent and trustworthy enough to influence business decisions.
The context layer had to stay fresh as schemas and business logic changed.

Architecture

Spotify built Vedder as a ReAct-style data assistant backed by human-owned context clusters. Each cluster contains curated datasets, vetted question-SQL pairs, and business docs. The agent selects an appropriate cluster, generates and executes SQL against the warehouse, returns the answer with query and sources, then logs conversations and health signals back to domain owners for maintenance.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-access-surfaces["user access surfaces"] --> vedder-react-data-agent["Vedder ReAct data agent"]
  vedder-react-data-agent --> cluster-context-layer["cluster context layer"]
  cluster-context-layer --> dataset-context["dataset context"]
  cluster-context-layer --> vetted-question-sql-pairs["vetted question-SQL pairs"]
  cluster-context-layer --> business-docs["business docs"]
  vedder-react-data-agent --> data-warehouse-query-tool["data warehouse query tool"]
  vedder-react-data-agent --> conversation-feedback-log["conversation feedback log"]
  conversation-feedback-log --> cluster-owner-dashboard["cluster owner dashboard"]
  conversation-feedback-log --> cluster-health-monitor["cluster health monitor"]
  cluster-health-monitor --> cluster-owner-dashboard
  cluster-owner-dashboard --> cluster-context-layer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`user access surfaces`	Slack bot, MCP server, and web UI where users ask questions and inspect results.	`Vedder ReAct data agent`
`Vedder ReAct data agent`	Selects context, reasons and acts over tool calls, writes SQL, runs it, and returns answer, query, and sources.	`user access surfaces`, `cluster context layer`, `data warehouse query tool`, `conversation feedback log`
`cluster context layer`	Domain-owned context package composed of relevant datasets, vetted examples, and docs.	`Vedder ReAct data agent`, `dataset context`, `vetted question-SQL pairs`, `business docs`, `cluster owner dashboard`
`dataset context`	Provides relevant tables with schema, profiling, cardinality, common values, and partitions.	`cluster context layer`
`vetted question-SQL pairs`	Few-shot examples curated or approved by domain experts to teach canonical query patterns.	`cluster context layer`
`business docs`	Captures terminology, gotchas, definitions, and column-use guidance that schemas do not encode.	`cluster context layer`
`data warehouse query tool`	Executes generated SQL against Spotify's warehouse.	`Vedder ReAct data agent`
`conversation feedback log`	Records conversations, SQL, answers, and user feedback for owners to improve clusters.	`Vedder ReAct data agent`, `cluster owner dashboard`, `cluster health monitor`
`cluster health monitor`	Calculates health signals such as data health, pair validity, coverage, and reproducibility.	`conversation feedback log`, `cluster owner dashboard`
`cluster owner dashboard`	Shows experts where to spend curation effort and suggested actions when cluster health degrades.	`cluster context layer`, `conversation feedback log`, `cluster health monitor`

Data flow

A user asks a data question in Slack, an MCP-enabled IDE/tool, or the web UI. Vedder selects the relevant domain cluster, uses curated datasets, examples, and docs to generate SQL, runs the query against the warehouse, returns the answer with sources, and logs questions, SQL, feedback, and health signals to cluster owners.

“When a question comes in, the agent picks the appropriate context, writes the SQL query, runs it against our warehouse, and returns the answer alongside the query and its sources.”article, stated

“Vedder logs every conversation and query, and the questions, answers, generated SQL and user feedback are shown to cluster owners.”article, stated

Why this architecture

Spotify deliberately placed a human-owned context layer between the LLM and the warehouse because the warehouse is too large for raw-schema prompting, and schema alone cannot carry business meaning. Domain experts decide what context is true; the model reasons over that curated context.

“We needed something in between. A layer that captures what actually matters about a slice of the warehouse, owned by people who own and understand the domain.”article, stated

“The model reasons over context. It doesn't decide what's true about the data, the experts do. This isn’t about replacing the people that they know the best how to work with our data, it’s about giving them more leverage. Shipping their expertise in a more”article, stated

Models used: LLM data agent · ReAct reasoning loop

Scale

Spotify's data assistant operates over more than 70,000 datasets containing petabytes of data.
“with over 70,000 datasets at Spotify, amounting to petabytes of data”article, stated
Spotify's data platform processes 1.4 trillion data points daily.
“Spotify's data platform processes 1.4 trillion data points daily across 70,000+ datasets.”article, stated
Since August 2025, Vedder has been used by more than 2,100 Spotifiers.
“It has been actively utilized since August 2025 by over 2,100 Spotifiers”article, stated
The assistant has handled more than 13,000 conversations and 60,000 messages.
“within 13,000+ conversations, and 60,000+ messages using 177 clusters”article, stated
Vedder's context layer currently covers 177 clusters across many business domains.
“using 177 clusters covering advertising, podcasts, music, audiobooks, finances, creators' tools, and more than a dozen other fields.”article, stated
Only 12.5% of automatically proposed question-SQL pairs were accepted by curators.
“They accepted only 12.5% of the proposed pairs.”article, stated

Alternatives considered (roads not taken)

put all warehouse schemas into the LLM

Rejected because: Context windows cannot fit the full warehouse, and schemas alone miss business semantics and legacy edge cases.

“Spotify, amounting to petabytes of data, no single individual can claim knowledge of everything. Just putting all schemas into an LLM doesn’t work at this scale. Data Platform

”article, stated

“schemas do not convey all the information. If a column has the INT64 type, then it doesn’t say anything about how those less than 100 are legacy test data”article, stated

skip curation and generate examples from query history

Rejected because: Most query-history-derived pairs were noisy, one-off, wrong-table, or poor canonical examples.

“The obvious shortcut was to skip the curator. Our data warehouse holds the complete query history of every data expert who has ever used it.”article, stated

“The other 87.5% were ad-hoc exploration, debugging sessions, one-off answers no one would ask again, queries that used the wrong table”article, stated

continue routing every question to individual Slack experts

Rejected because: Demand from thousands of teams had outpaced what individual experts could handle one request at a time.

“You'd message the corresponding data expert on Slack, wait until they had time to help. But with thousands of teams moving fast”article, stated

Patterns

`tool-use-orchestration` (agent-architecture)

Vedder uses a ReAct loop to pick context, call tools, write SQL, execute it, and adjust from tool responses.

“It follows a ReAct [1] loop, reasoning and acting in steps, adjusting based on what each tool call returns.”article, stated

`rag-retrieval` (context-management)

The assistant retrieves domain-specific context clusters rather than prompting over the whole warehouse.

“At Spotify, we call data domains, clusters. Those domains can be tied to an initiative, an organization, or an adhoc interest.”article, stated

`human-in-the-loop-gate` (governance)

Domain experts curate the context and approve canonical question-SQL examples before they influence the model.

“A domain expert writes or approves each pair, picking examples that teach the patterns they'd want a colleague to follow.”article, stated

`data-quality-loop` (operations)

Every Vedder conversation feeds back to cluster owners, and health scores guide ongoing context maintenance.

“Every conversation with Vedder becomes a data point that feeds back into the system.”article, stated

“Data experts see the score and the underlying signals on their cluster dashboard, and use them to decide where to spend curation time.”article, stated

`eval-driven-development` (quality)

Cluster health metrics track whether curated examples remain valid, context covers real questions, and generated SQL is reproducible.

“How many of its curated pairs are still valid after recent schema changes? If a column gets renamed, the pairs referencing it degrade immediately.”article, stated

Tradeoffs

Put ownership of truth in curated clusters rather than in the model.

Chose domain-expert-owned datasets, examples, and docs over letting the LLM infer truth directly from warehouse schemas.

Rationale: The LLM can reason over context, but experts know which context is correct and canonical.

“The model reasons over context. It doesn't decide what's true about the data, the experts do.”article, stated

Require human review of examples despite having complete query history.

Chose curator-approved question-SQL pairs over automatically generated pairs from historical queries.

Rationale: The auto-generated pool was mostly noise, so review cost bought trustworthiness.

“They accepted only 12.5% of the proposed pairs. The other 87.5% were ad-hoc exploration, debugging sessions, one-off answers no one would ask again, queries that used the wrong table, or queries that were technically correct but taught the wrong pattern.”article, stated

Expose answers with query and sources instead of only final results.

Chose transparent answer surface over opaque assistant answers.

Rationale: Users can inspect how the answer was produced, which supports trust in the system.

“You can read how the result was produced, not just what it was. We built into the surfaces people already work: a Slack bot for quick questions while chatting on a thread, an MCP server for IDEs and AI tools, and a dedicated web UI for interactive exploration.”article, stated

Gotchas (what bit them)

Raw schemas made the model confidently wrong.

Root cause: Schemas omit semantics such as legacy data ranges, definition differences, and what concepts like active user mean.

Resolution: Use curated docs, dataset profiling, and canonical examples owned by domain experts.

“Provide the same number of tables to a model, and it will be confident in selecting the wrong one.”article, stated

“Docs: additional business context. This could be terminology, gotchas, definitions that vary by team, which columns to use and which to avoid.”article, stated

Automatically mined query history looked scalable but mostly produced bad teaching examples.

Root cause: Historical queries include exploration, debugging, one-off answers, wrong tables, and technically correct but misleading patterns.

Resolution: Require domain experts to review and mark examples as canonical.

“Query history is rich. Most of it is noise. And the signal doesn't label itself.”article, stated

“We wanted the examples that would influence the assistant’s behavior to be reviewed and marked as canonical by those familiar with the data.”article, stated

The context layer can decay as data and business logic change.

Root cause: Schemas evolve, columns are renamed, tables are deprecated, and definitions shift over time.

Resolution: Continuously calculate cluster health and surface degraded context to owners with suggested actions.

“Data changes, business logic shifts, and context that was accurate last month can be wrong today.”article, stated

“If any of these degrade, then the cluster’s health score reflects that and actions are suggested.”article, stated

Evaluation

Spotify evaluated trust and usefulness through curator acceptance of generated question-SQL pairs, production usage, cluster health scores, reproducibility checks, and user feedback loops.

Metrics: 12.5% curator acceptance rate for proposed pairs · 2,100+ users · 13,000+ conversations · 60,000+ messages · 177 clusters · cluster health signals including pair validity, coverage, and SQL reproducibility

“How well does the context cover questions people are actually using? How reproducible was the generated SQL? And a handful of others.”article, stated

“Every question-SQL pair they approve, every doc they clarify helps the next users get even more accurate insights.”article, stated

Outcomes

Spotify launched Vedder across Slack, MCP/IDE tools, and a web UI.
More than 2,100 Spotifiers used it in 13,000+ conversations and 60,000+ messages.
The system made domain experts' knowledge reusable through curated clusters rather than one-off support.

Vedder runs over 177 domain clusters in a data environment of 70,000+ datasets and petabytes of data.

“We built into the surfaces people already work: a Slack bot for quick questions while chatting on a thread, an MCP server for IDEs and AI tools, and a dedicated web UI”article, stated

“They spent less time answering one-off questions, more time shaping the knowledge layer that answers thousands.”article, stated