Scaling Enterprise RAG with Advanced Vector Search Migration

danswer · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise RAG search infrastructure

Danswer needed to scale enterprise RAG search across organizational knowledge sources when its previous vector database could not support hybrid ranking, recency decay, or multi-vector document indexing.

Constraints:

• Enterprise search needs team-specific names and terms that pure vector search misses.
• Internal documents can have stale duplicates with conflicting information.
• Danswer is largely self-hosted, so resource efficiency matters for customers without large cloud budgets.

Architecture

Danswer migrated its core enterprise RAG retrieval layer to Vespa. Vespa enabled normalized hybrid search across vector and keyword signals, flexible time-decay ranking functions over document freshness, and multi-vector-per-document indexing so different section sizes can be searched without duplicating whole documents.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Enterprise knowledge sources"] --> B["Danswer ingestion"]
  B --> C["Vespa index"]
  C --> D["Hybrid search ranking"]
  C --> E["Multi-vector document representation"]
  D --> F["RAG context retriever"]
  E --> F
  F --> G["LLM answer layer"]

Components

Data flow

Danswer ingests organizational knowledge sources such as Google Drive, Slack, and Salesforce, indexes documents with hybrid keyword/vector representations and multiple embeddings per document in Vespa, ranks results with custom boost/decay functions, and passes retrieved context to the LLM for a search/chat interface.

Why this architecture

Danswer moved to Vespa because enterprise RAG needed more than pure vector search: team-specific keywords, stale-document decay, multiple vectors per document, and internet-scale search features that still worked for self-hosted deployments.

Models used: LLMs for chat/search answers · vector embedding models · late-interaction models such as ColBERT

Scale

• Danswer connects sources such as Google Drive, Slack, and Salesforce.
  “We connect all of the disparate knowledge sources of a team (like Google Drive, Slack, Salesforce, etc.)”article, stated
• The search layer had to scale past several million documents.
  “these workarounds often suffered in accuracy once the scale of documents increased past several million.”article, stated
• Danswer served scales up to tens of millions of documents per customer.
  “we were serving scales of up to tens of millions of documents per customer but Vespa was built for internet scales of data”article, stated

Alternatives considered (roads not taken)

Vector-only search

Rejected because: It missed team-specific terms that had no general English semantic representation.

Separate keyword and vector search engines

Rejected because: Separate systems caused weighting issues because signals could not be considered together.

Post-processing time decay after initial search

Rejected because: The workaround lost accuracy once document scale reached several million.

Patterns

rag-retrieval (retrieval)

Danswer retrieves enterprise context before passing it to the LLM.

hybrid-search (retrieval)

Vespa normalizes and combines keyword and vector search signals.

retrieval-ranking (retrieval)

Danswer uses custom ranking functions including time-decay relevance over document freshness.

cost-aware-routing (operations)

Multi-vector-per-document indexing reduced resource requirements for self-hosted customers.

Tradeoffs

Search engine

Chose Vespa over previous vector DB and separate keyword/vector engines.

Rationale: Vespa supports normalized hybrid search, custom ranking, multiple vectors per document, and internet-scale operation.

Document representation

Chose multiple vector embeddings per document over duplicating documents for every section/chunk size.

Rationale: The multi-vector approach captures both broad and detailed context while reducing serving resources.

Self-hosted search platform

Chose resource-efficient index design over resource-heavy duplication.

Rationale: Self-hosted customers may lack powerful servers or expensive cloud budgets.

Gotchas (what bit them)

Pure vector search missed important internal terms.

Root cause: Names like internal project or team terms had no useful general-English semantic representation.

Resolution: Add keyword search and then migrate to Vespa to combine keyword and vector signals cleanly.

Stale duplicate documents created conflicting answers.

Root cause: Internal documents are not always cleaned up, leaving multiple versions with conflicting information.

Resolution: Apply flexible time-based decay ranking at search time.

Search flexibility increased system complexity.

Root cause: Vespa exposes rich configuration, deployment, and query/indexing options.

Resolution: Use Vespa for core retrieval while moving Danswer Cloud toward Vespa Cloud to offload operational complexity.

Evaluation

Danswer evaluated the migration through search accuracy, ability to combine keyword/vector signals, ranking flexibility, scale past millions of documents, and resource requirements for self-hosted deployments.

Metrics: search accuracy · hybrid normalization quality · ranking-function flexibility · document-scale accuracy · resource footprint

Outcomes

• Danswer achieved normalized hybrid search across keyword and vector signals.
• The team implemented time-decay ranking for stale documents.
• Multi-vector-per-document indexing reduced resource needs for self-hosted customers.
• Vespa supported scales up to tens of millions of documents per customer.

The source is a vendor/platform article, but it names the migration tradeoffs and feature requirements clearly.