Building Production-Scale AI Search with Knowledge Graphs, MCP, and DSPy

dropbox · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise AI search and agent context over fragmented work content

Dropbox needed Dash to search, answer, and run agentic queries over proprietary work content scattered across many SaaS applications, while staying fast, ACL-compliant, relevant, and practical within LLM context-window limits.

Constraints:

• Work content lives in third-party SaaS walled gardens rather than in public model pretraining data.
• Connectors must handle each application's rate limits, API quirks, ACLs, and permissions.
• Content types include documents, images, PDFs, audio, and videos with different understanding requirements.
• Federated retrieval is easy to start but inherits variable API speed, ranking quality, and limited shared-content access.
• MCP tool definitions and retrieval results can consume too much context window space.
• Prompt and judge quality must scale across dozens of prompts and many engineers.

Architecture

Dropbox Dash is built around a context engine. Custom connectors crawl third-party apps, normalize content into formats such as markdown, extract metadata and embeddings, understand multimodal media where needed, model cross-app relationships into graph-derived knowledge bundles, store chunks, embeddings, and contextual graph representations in secure lexical/vector stores, and apply ranking plus ACL personalization. For agents, Dropbox wraps indexes as MCP-style super tools, stores large tool results locally rather than in LLM context, routes complex queries to narrower sub-agents, evaluates retrieval relevancy with LLM judges aligned to humans, and uses DSPy to optimize judge and platform prompts across the Dash stack.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  work-user[Work user] --> dropbox-dash[Dropbox Dash]
  dropbox-dash --> context-engine[Context engine]
  dropbox-dash --> agentic-query-layer[Agentic query layer]
  third-party-apps[Third-party apps] --> connectors[Connectors]
  connectors --> content-normalization[Content normalization]
  content-normalization --> content-understanding[Content understanding]
  content-understanding --> knowledge-graph-builder[Knowledge graph builder]
  knowledge-graph-builder --> people-model[People model]
  knowledge-graph-builder --> knowledge-bundles[Knowledge bundles]
  content-understanding --> secure-data-stores[Secure data stores]
  knowledge-bundles --> lexical-bm25-index[Lexical BM25 index]
  knowledge-bundles --> vector-store[Vector store]
  secure-data-stores --> lexical-bm25-index
  secure-data-stores --> vector-store
  lexical-bm25-index --> ranking-pipeline[Ranking pipeline]
  vector-store --> ranking-pipeline
  people-model --> ranking-pipeline
  ranking-pipeline --> dropbox-dash
  context-engine --> mcp-super-tool[MCP super tool]
  mcp-super-tool --> lexical-bm25-index
  mcp-super-tool --> vector-store
  mcp-super-tool --> local-tool-result-store[Local tool-result store]
  agentic-query-layer --> mcp-super-tool
  agentic-query-layer --> sub-agent-classifier[Sub-agent classifier]
  sub-agent-classifier --> narrow-sub-agents[Narrow sub-agents]
  ranking-pipeline --> llm-judge[LLM judge]
  human-evaluators[Human evaluators] --> llm-judge
  llm-judge --> dspy-optimizer[DSPy optimizer]
  dspy-optimizer --> prompt-stack[Prompt stack]
  prompt-stack --> agentic-query-layer

Components

Data flow

Dash ingests content from third-party apps through custom connectors, normalizes files into markdown-like formats, extracts metadata, links, titles, embeddings, and multimodal understanding, then models relationships across meetings, people, documents, transcripts, and projects. Embeddings, chunks, and graph-derived knowledge bundles flow into secure lexical and vector stores. Query-time ranking personalizes and ACL-filters retrieved results. Agentic queries call the context engine through consolidated MCP super tools, keep large tool results out of the LLM context, and route complex tasks to narrower sub-agents. Retrieval quality is measured with human labels, LLM judges, NDCG, prompt tuning, and DSPy optimization.

5 supporting quotes

“It connects to all your third-party apps and brings it into one place, so you can search, get answers, and do the agentic queries”article, stated

“we normalize a lot of the different files that come in and get it into a format like markdown. Then, we’re looking at extracting key information. We're going to be looking at titles, metadata, trying to extract links, and generate different embeddings. For”article, stated

“all that information (embeddings, chunks, contextual graph representations) flows into our highly secure data stores”article, stated

“we apply multiple ranking passes on any retrieved results so they are personalized and ACL’d to you. Altogether,”article, stated

“around a tool. Let's call it a super tool. And so instead of 5-10 different retrieval tools, we just have one. This helps a ton with cleaning things up overall. * Modeling data within knowledge graphs can significantly cut our token usage as well, because you're really just getting”article, stated

Why this architecture

Dropbox chose a heavy index/context-engine architecture because Dash needs company-wide shared content, enrichment at ingestion time, offline ranking experiments, fast query performance, ACL enforcement, graph-derived context, and controllable agent context windows. Federated retrieval was easier to start, but the team judged it too dependent on heterogeneous APIs and too costly in tokens for complex MCP/chatbot usage.

Models used: LLMs · CLIP-based models · Multimodal models · OpenAI o3 · DSPy-optimized models · Planning LLM · Sub-agent models

Scale

• The motivating user context is around 50 open tabs and at least 50 SaaS accounts.
  “I probably have about 50 tabs open right now—and at least another 50 accounts for other SaaS apps.”article, stated
• Dash caps agent context around 100,000 tokens.
  “with Dash, we're trying to cap things to about 100,000 tokens.”article, stated
• Simple MCP agent queries can take up to 45 seconds, while raw indexes return content within seconds.
  “even a simple query can take up to 45 seconds—whereas with the raw index, you're getting all the content coming back very quickly, within seconds.”article, stated
• The super-tool pattern replaces 5-10 retrieval tools with one tool.
  “instead of 5-10 different retrieval tools, we just have one.”article, stated
• The first LLM-judge prompt had 8% disagreement with human labels.
  “The first prompt for our judge wasn’t bad—8% disagreed—but the lower, the better.”article, stated
• Dash has over 30 prompts across its stack.
  “We have over 30 different prompts today throughout the Dash stack”article, stated
• At any time, 5 to 15 engineers may be tweaking prompts.
  “at any given time we might have 5 to 15 different engineers tweaking these prompts”article, stated
• The described techniques were built over the last few years by a large engineering team.
  “over the last few years with a big engineering team working on this day-in and day-out.”article, stated

Alternatives considered (roads not taken)

Federated retrieval

Rejected because: It is easy to start and fresh, but depends on varied API speed, quality, ranking, and access limits, and requires heavy query-time post-processing.

Graph database for knowledge graphs

Rejected because: Latency, query patterns, and hybrid retrieval integration were challenging.

Many separate MCP retrieval tools

Rejected because: Tool definitions consume valuable context-window space and degrade chat/agent effectiveness.

Put full tool results into the LLM context

Rejected because: Tool results can be huge, so Dropbox stores them locally and avoids filling the context window.

Manual prompt whack-a-mole

Rejected because: With dozens of prompts and many engineers, hand-editing strings for edge cases breaks at scale.

Patterns

rag-retrieval (context-engineering)

Dash builds a context engine that retrieves proprietary work content and graph context for search, answers, and agentic queries.

“ Altogether, this is what we call our context engine. And once you have that, you can introduce APIs on top of it and [build entire products like”article, stated

“It connects to all your third-party apps and brings it into one place”article, stated

hybrid-search (retrieval)

Dropbox combines BM25 lexical retrieval with dense vector storage and hybrid ranking.

retrieval-ranking (retrieval)

The system applies multiple personalized, ACL-aware ranking passes and uses NDCG and judge scores to improve relevance.

llm-as-judge (evaluation)

Dropbox scores retrieved information for relevance with an LLM judge and iteratively reduces disagreement with human labels.

tool-use-orchestration (agent-architecture)

Dash wraps the index as a super tool and uses classifiers/sub-agents to keep each agent's tool set narrow.

context-compaction (context-engineering)

Dropbox reduces context pressure by graph-bundling relationships, consolidating tools, and keeping large tool results out of the LLM context.

eval-driven-development (evaluation)

Human labels, judge disagreement, prompt tuning, model upgrades, and DSPy optimization form a measurement-driven retrieval-improvement loop.

model-routing (model-operations)

The platform expects different planning and sub-agent models, with DSPy helping regenerate prompts for model switching.

Tradeoffs

Choose index-based retrieval

Chose Pre-processing content at ingestion time over Federated query-time retrieval.

Rationale: Indexing is much more work, but it enables company-wide content, enriched datasets, offline ranking experiments, and fast query-time retrieval.

Use BM25 alongside vectors

Chose Lexical BM25 plus dense vector storage over Pure vector-only retrieval.

Rationale: Hybrid retrieval is supported, but Dropbox found BM25 alone highly effective with the right signals.

Represent graph context as knowledge bundles

Chose Asynchronous graph-derived bundles passed through the index pipeline over Serving directly from a graph database.

Rationale: The bundle approach avoided graph database latency and hybrid-query challenges while still adding relationship context.

Consolidate MCP retrieval

Chose One index-backed super tool and narrow sub-agents over Many retrieval tools exposed directly to the LLM.

Rationale: This reduces tool-definition context overhead and gives complex queries a smaller, more relevant tool set.

Let the judge retrieve context too

Chose RAG as a judge over A judge relying only on precomputed or pretrained knowledge.

Rationale: Workplace acronyms and local context may be unknown to the judge, so it sometimes needs retrieval before scoring relevance.

Use DSPy for prompt optimization

Chose Programmatic prompt optimization over evals over Manual prompt-string editing for every edge case and model switch.

Rationale: DSPy improved LLM-judge results, helped manage dozens of prompts, and makes model switching faster.

Gotchas (what bit them)

LLMs cannot answer work questions if proprietary content is outside their reach.

Root cause: Work content is scattered across walled gardens and SaaS applications.

Resolution: Use connectors and a context engine to bring third-party work content into one searchable, ACL-aware place.

Connectors are not commodity plumbing.

Root cause: Every app has its own rate limits, API quirks, ACLs, and permissions.

Resolution: Invest in custom crawlers and connector engineering before relying on downstream retrieval quality.

MCP can rot the context window.

Root cause: Tool definitions and returned retrieval content quickly consume context budget.

Resolution: Cap context, consolidate retrieval into super tools, use graph context, store large tool results locally, and route to narrow sub-agents.

Graph databases were not the practical retrieval substrate.

Root cause: Latency, query patterns, and hybrid retrieval were hard to make work.

Resolution: Build graph relationships asynchronously into knowledge bundles that flow through the same index pipeline.

Human click signals disappear when retrieval serves an LLM instead of a person.

Root cause: Chat systems retrieve context for the model, so humans no longer directly click retrieved documents to indicate relevance.

Resolution: Use LLM-as-judge plus human evaluator labels to score relevancy and train improvement loops.

Prompt editing becomes whack-a-mole at stack scale.

Root cause: Many prompts and many engineers create regressions when edge-case fixes are hardcoded as text strings.

Resolution: Use programmatic prompt optimization with DSPy and eval-defined objectives.

Evaluation

Dropbox evaluates retrieval and agent context quality through NDCG scoring, human document labels, LLM-as-judge relevancy scoring, disagreement analysis, prompt tuning, model upgrades such as OpenAI o3, RAG-assisted judges, and DSPy prompt optimization. Prompt optimizers are especially applied where rubrics and eval outcomes are crisp, such as LLM-as-judge.

Metrics: NDCG · Human/LLM judge disagreement rate · One-to-five relevance scores · Human thumbs up/down feedback · Human evaluator labels · Prompt optimization eval outcomes · Model-upgrade judge agreement · Retrieval recall · Context-window token usage · Query latency

Outcomes

• Dropbox built Dash as a universal work search and agentic-query product over Dropbox and third-party apps.
• Index-based retrieval was judged superior at scale despite large engineering and storage costs.
• BM25 proved highly effective and became a strong indexing workhorse alongside vector storage.
• People modeling and knowledge-graph context produced nice wins in NDCG/relevance.
• Super tools, graph context, local result storage, and sub-agent selection reduced MCP/context-window pressure.
• The first judge prompt had 8% human disagreement, and prompt tuning, o3, RAG-as-judge, and DSPy reduced disagreements further.
• DSPy is being expanded across over 30 Dash prompts spanning ingest, judges, offline evals, and online agentic platform prompts.

The source reports 50+ tabs and 50+ SaaS accounts as the motivating user problem, a roughly 100K-token context cap, simple MCP queries taking up to 45 seconds versus raw index retrieval within seconds, consolidation from 5-10 retrieval tools to one super tool, 8% initial judge disagreement, over 30 prompts, 5-15 engineers editing prompts, and several years of work by a large engineering team.