Real-time Question-Answering System with Two-Stage LLM Architecture for Sales Content Recommendations

microsoft · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Sales copilot content recommendation and question answering

Microsoft sellers needed relevant sales documentation from Seismic inside MSX Sales Copilot quickly enough to use during calls or preparation, despite lacking programmatic access to full document content.

Constraints:

• The system needed real-time inference and top document recommendations in seconds.
• Document content was not programmatically available, so the team had to rely on metadata and prompt engineering.
• A single-stage retriever was fast but missed nuanced query-document interactions.
• Ground-truth relevance labels were limited, constraining traditional supervised ranking metrics.

Architecture

The system uses a two-stage retrieval architecture deployed as an Azure Machine Learning endpoint and invoked from MSX Sales Copilot through Semantic Kernel. A DistilBERT bi-encoder embeds seller queries and metadata-engineered document prompts for fast top-100 retrieval; a MiniLM cross-encoder then re-ranks query-document pairs and returns the top five sales documents.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["MSX Sales Copilot"] --> B["Semantic Kernel planner"]
  B --> C["AML recommendation endpoint"]
  D["Seismic metadata catalog"] --> E["Metadata prompt builder"]
  E --> F["Offline document embeddings"]
  J["Weekly metadata refresh"] --> D
  J --> F
  C --> G["Bi-encoder retriever"]
  F --> G
  G --> H["Cross-encoder reranker"]
  H --> I["Top-5 recommendation results"]
  I --> A

Components

Data flow

Seismic metadata is transformed into natural-language document prompts and embedded offline by the bi-encoder. At query time, Semantic Kernel routes a seller request to the AML endpoint, the query is embedded, cosine similarity retrieves top candidates, the cross-encoder re-ranks the top 100 query-document pairs, and the Copilot interface returns the top five documents.

Why this architecture

The two-stage design balances latency and relevance: precomputed bi-encoder embeddings make retrieval fast, while the cross-encoder adds query-document interaction only on a narrowed candidate set. Metadata prompt engineering compensates for the lack of direct document-content access.

Models used: DistilBERT MS MARCO bi-encoder · MS MARCO MiniLM cross-encoder

Scale

• The system returns recommendations in a few seconds.
  “returns the most relevant content recommendations in just a few seconds even for large datasets.”article, stated
• The re-ranker receives the top 100 candidate documents.
  “The bi-encoder retrieves the top-100 candidate documents based on cosine similarity with the query embedding.”summary, stated
• The end result returns the top five relevant documents.
  “When the planner determines that a content recommendation is needed, it invokes the AML endpoint, which processes the query through both stages and returns the top-5 most relevant documents.”summary, stated
• Human evaluation covered 31 queries with four domain experts.
  “Four human domain experts rated the top-5 results for 31 evaluation queries on a 0-5 scale”summary, stated
• GPU median latency was 0.54 seconds with batch size 2, compared with 1.49 seconds on a DS4 v2 CPU machine.
  “With b=2, the GPU-enabled VM achieved median latency of 0.54 seconds, while the DS4 v2 CPU machine achieved 1.49 seconds.”summary, stated

Alternatives considered (roads not taken)

Use a single-stage bi-encoder retriever only

Rejected because: The full two-stage architecture produced more relevant recommendations for most evaluated queries.

Pass too few candidates to the cross-encoder

Rejected because: Small K values lacked enough diversity for the reranker.

Pass too many candidates to the cross-encoder

Rejected because: Large K values created unacceptable latency.

Search full document content directly

Rejected because: Programmatic access to actual document content was not available at the time.

Patterns

rag-retrieval (retrieval)

The recommender matches seller questions to relevant sales documents using bi-encoder retrieval and cross-encoder re-ranking.

cost-aware-routing (operations)

Offline document embeddings, top-K retrieval, and tuned cross-encoder batch size reduce real-time compute and latency.

eval-driven-development (quality)

The team used human annotation, ablation studies, latency tests, and seller survey feedback to validate retrieval quality and production usefulness.

prompt-versioning (context)

Document metadata was engineered into natural-language prompts so the embedding model could operate on structured features.

Tradeoffs

Use a bi-encoder followed by a cross-encoder.

Chose Two-stage retrieval and re-ranking over Single-stage retrieval.

Rationale: The bi-encoder gives speed through cached embeddings; the cross-encoder adds pairwise relevance for the narrowed set.

Represent numerical metadata as categorical labels in prompts.

Chose Percentile-mapped high/medium/low/zero labels over Raw numerical metadata in document prompts.

Rationale: Natural-language prompts fit the embedding model better than raw metrics while preserving engagement signals.

Refresh metadata weekly rather than continuously.

Chose Weekly refresh cycle over Real-time document metadata updates.

Rationale: A weekly cadence kept the catalog current enough while avoiding a real-time update pipeline.

Gotchas (what bit them)

No direct document-content access forced a metadata-only recommender.

Root cause: The Seismic document content was not programmatically accessible.

Resolution: The team converted about 20 metadata features into English-formed document prompts for embedding.

The cross-encoder improves relevance but can dominate latency if the candidate set is too large.

Root cause: Cross-encoder inference must process query-document pairs at request time.

Resolution: The team tuned K and found K=100 balanced recall and latency.

Verbose or generic seller queries hurt recommendation relevance.

Root cause: The semantic matching setup struggled when queries lacked focused retrieval intent.

Resolution: Human evaluation identified this failure mode for future refinement.

Evaluation

Microsoft used human expert ratings, ablation against a retriever-only baseline, latency tests across Azure VM configurations, and post-launch seller feedback.

Metrics: Top-5 relevance score from four human experts · 15/31 relevant queries · 9/31 somewhat relevant queries · 7/31 not relevant queries · 90% of queries improved by full two-stage architecture · Median latency by VM/batch size · Task relevance rating 4/5 · Document relevancy rating 3.7/5

Outcomes

• The recommender was deployed in production MSX Sales Copilot.
• It returns top-five document recommendations in seconds.
• Human evaluation favored the two-stage architecture over retriever-only retrieval.
• Seller feedback rated task relevance 4/5 and document relevance 3.7/5.
• The integrated Copilot experience improved on the previous external filter-based search.

Production deployment in MSX Sales Copilot; top-100 retrieval, top-5 results, 31-query expert evaluation, K=100 reranking, weekly metadata refresh, GPU median latency of 0.54s at batch size 2.