Scaling AI-Powered File Understanding with Efficient Embedding and LLM Architecture

dropbox · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-powered file summarization and question answering

Dropbox needed to make summaries and Q&A work inside web file previews across many file types and multiple files, while controlling latency, cost, hallucinations, privacy, and context selection.

Constraints:

• Dropbox supports about 300 file types and massive daily file-conversion volume.
• LLMs require text context, but files may be documents, video, audio, CAD, or other complex formats.
• Summaries need broad topic coverage rather than repeated local summaries.
• Q&A needs query-relevant chunks and dynamic context sizing across multiple selected files.
• Privacy requires users to choose which files are shared with the LLM.
• Embedding and LLM calls must be cached or bounded to reduce cost and latency.

Architecture

Dropbox extended Riviera, its file-conversion framework, so text extraction, embeddings, summarization, and Q&A are treated as conversion plugins. Riviera routes requests through jailed plugins and caches intermediate states. Text is chunked into paragraphs and embedded; summarization uses k-means clustering to choose semantically diverse representative chunks; Q&A embeds the user question and selects chunks by similarity. Multi-file Q&A uses relevance-score spread over top chunks to choose context, while real-time processing preserves user-controlled file sharing.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["File preview surface"] --> B["Riviera conversion framework"]
  B --> C["Jailed plugins"]
  B --> D["Text extraction plugin"]
  D --> E["Embedding plugin"]
  E --> F["Riviera cache"]
  E --> G["Summary plugin"]
  E --> H["Q&A plugin"]
  H --> I["Multi-file selector"]
  G --> J["LLM"]
  H --> J

Components

Data flow

A file preview request enters Riviera, which converts files to browser-consumable forms and raw text through isolated plugins. The embedding plugin chunks text and produces embeddings that are cached like other conversions. Summaries select dissimilar representative chunks via k-means and make one LLM call. Q&A embeds the user question, ranks chunks by vector similarity, selects relevant chunks and files, then sends the query and context to the LLM. Follow-up questions are requested via function calling and structured outputs.

Why this architecture

Dropbox reused Riviera because file preview conversion, plugin isolation, conversion graphs, and caching already existed at Dropbox scale. Treating embeddings as conversions lets summaries and Q&A share cached chunks and embeddings. K-means provides diverse summary context with a single LLM call, while similarity ranking and power-law cutoffs keep Q&A context focused. Real-time LLM calls preserve file-sharing control for users.

Models used: large language models · embedding models · k-means clustering · function calling · structured outputs

Scale

• Riviera supports conversions between about 300 file types.
  “We currently support conversions between about 300 file types”article, stated
• Riviera processes about 2.5 billion requests per day.
  “the system crunches through about 2.5 billion requests—totalling nearly an exabyte, or one billion gigabytes of data—per day.”article, stated
• Riviera processes nearly an exabyte of data per day.
  “totalling nearly an exabyte, or one billion gigabytes of data—per day.”article, stated
• K-means summaries covered about 50% more topics than map-reduce.
  “the summaries covered a broader range of topics—approximately 50% more than with map-reduce”article, stated
• Multi-file relevance selection uses the top 50 text chunks and cuts the bottom 20% of the score spread.
  “takes the max and min relevance scores from the top 50 text chunks related to the user query, as calculated through the embeddings, and cuts off the bottom 20% of that spread.”article, stated
• Cost-per-summary dropped 93% and cost-per-query dropped 64%.
  “Cost-per-summary dropped by 93%, and cost-per-query dropped by 64%.”article, stated
• P75 summary latency dropped from 115 seconds to 4 seconds.
  “The p75 latency of summaries decreased from 115 seconds to 4 seconds”article, stated
• P75 query latency dropped from 25 seconds to 5 seconds.
  “the p75 latency for queries decreased from 25 seconds to 5 seconds.”article, stated

Alternatives considered (roads not taken)

Summary-of-summaries / map-reduce

Rejected because: It repeated similar information, lost content, and compounded hallucination risk across multiple LLM calls.

Send the entire file directly through multiple summary calls

Rejected because: Each LLM call creates another chance for hallucination and makes errors harder to pinpoint.

Precompute embeddings and AI responses ahead of time

Rejected because: Real-time processing lets users choose exactly which files are shared with the LLM and avoids unused precompute cost.

Do not cache embeddings

Rejected because: The initial version made repeated LLM calls for the same document.

Patterns

rag-retrieval (retrieval)

Q&A embeds the query, retrieves similar text chunks, and sends selected context to the LLM for answering.

semantic-caching (optimization)

Dropbox caches transcripts and embeddings as Riviera conversion states so summaries and Q&A can reuse them.

retrieval-ranking (retrieval)

Multi-file Q&A ranks chunks by relevance scores and cuts off low-score portions of the spread.

structured-output-contract (engineering-practice)

Dropbox uses function calling and structured outputs to request follow-up questions alongside summaries or answers.

privacy-preserving-processing (governance)

Dropbox computes AI responses in real time so users decide which files are shared with the LLM.

latency-optimization (operations)

Embedding caching, context selection, and file-conversion reuse reduced p75 summary and query latency substantially.

Tradeoffs

Summary context selection

Chose k-means clustering for diverse representative chunks over map-reduce summary-of-summaries.

Rationale: K-means improved topic diversity and reduced hallucination opportunities by using one final LLM call.

AI response timing

Chose real-time responses over precomputing responses ahead of time.

Rationale: Real-time computation lets users control file sharing and avoids precomputing unused answers.

Embedding architecture

Chose embeddings as cacheable file conversions over embedding generation coupled directly to every summary or Q&A call.

Rationale: Separating embeddings from generation lets Dropbox reuse conversions and share the same cache across features.

Gotchas (what bit them)

Summary-of-summaries loses topic diversity.

Root cause: Intermediate summaries repeat similar information before the final merge.

Resolution: Dropbox uses k-means to select semantically dissimilar representative chunks.

Multiple LLM calls compound hallucination risk.

Root cause: Every LLM call in a map-reduce summary chain creates a new chance to hallucinate.

Resolution: K-means creates one representative context blob for a single summary-generation call.

One-file AI understanding did not meet multi-file user needs.

Root cause: Users often need answers across selected collections rather than one document.

Resolution: Dropbox expanded Riviera and context-selection algorithms for multi-file summary and Q&A.

Uncached embeddings wasted repeated work.

Root cause: The initial version repeatedly called the LLM for the same document.

Resolution: Dropbox cached embeddings and reused the same chunks for summaries and Q&A.

Evaluation

Dropbox evaluates the architecture through topic diversity, hallucination risk, cost, p75 latency, context relevance, and feature availability in early access.

Metrics: About 300 file types · About 2.5 billion requests per day · Nearly an exabyte per day · Approximately 50% broader topic coverage than map-reduce · Top 50 query-related chunks · Bottom 20% score-spread cutoff · 93% cost-per-summary reduction · 64% cost-per-query reduction · Summary p75 latency 115s to 4s · Query p75 latency 25s to 5s

Outcomes

• Dropbox shipped AI-powered summaries and Q&A in web file previews.
• The architecture expanded from one-file to multi-file understanding.
• Embedding caching reduced repeated work across summary and Q&A.
• K-means selection improved topic diversity and reduced hallucination risk relative to summary-of-summaries.
• Cost-per-summary dropped 93% and cost-per-query dropped 64%.
• P75 summary latency dropped from 115 seconds to 4 seconds, and p75 Q&A latency dropped from 25 seconds to 5 seconds.

Dropbox reports about 300 supported file types, about 2.5 billion conversion requests and nearly an exabyte per day, roughly 50% broader topic coverage with k-means than map-reduce, a top-50-chunk relevance selection, 93%/64% cost reductions, and p75 latency reductions from 115s to 4s and 25s to 5s.