Building Ask Learn: A Large-Scale RAG-Based Knowledge Service for Azure Documentation

microsoft · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Tech

Microsoft needed reliable, relevant, accurate, and verifiable answers over Microsoft Learn documentation for Microsoft Q&A and Copilot for Azure.

Constraints:

documentation corpus is large and continuously changing
naive RAG was not reliable enough for production
answers need verifiable grounding
retrieval chunks must balance context and token limits

Architecture

Microsoft Ask Learn is an advanced RAG knowledge service for Microsoft Learn content. A Knowledge Service chunks documentation, embeds chunks, stores vectors and metadata, exposes vector access through a web API, and feeds an orchestration layer with pre-retrieval, retrieval, post-retrieval, completion, post-completion, evaluation, and feedback-analysis services. Ask Learn then supports Microsoft Q&A Assist, Copilot for Azure grounding, and Copilot fallback/self-service scenarios.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  microsoft-learn-content["microsoft-learn-content"]
  knowledge-service["knowledge-service"]
  vector-database["vector-database"]
  orchestration-layer["orchestration-layer"]
  pre-retrieval-processing["pre-retrieval-processing"]
  post-retrieval-processing["post-retrieval-processing"]
  completion-service["completion-service"]
  evaluation-services["evaluation-services"]
  feedback-assessment-pipeline["feedback-assessment-pipeline"]
  microsoft-qa-assist["microsoft-qa-assist"]
  copilot-for-azure["copilot-for-azure"]
  microsoft-learn-content --> knowledge-service
  knowledge-service --> vector-database
  knowledge-service --> orchestration-layer
  vector-database --> orchestration-layer
  orchestration-layer --> pre-retrieval-processing
  orchestration-layer --> post-retrieval-processing
  orchestration-layer --> completion-service
  orchestration-layer --> evaluation-services
  orchestration-layer --> microsoft-qa-assist
  orchestration-layer --> copilot-for-azure
  post-retrieval-processing --> completion-service
  completion-service --> evaluation-services
  evaluation-services --> feedback-assessment-pipeline
  feedback-assessment-pipeline --> knowledge-service

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`microsoft-learn-content`	source documentation and training content used as RAG ground truth	`knowledge-service`
`knowledge-service`	chunks documentation, creates embeddings, stores vectors/chunks, keeps the vector database updated, and exposes access via API	`vector-database`, `orchestration-layer`
`vector-database`	stores embeddings, chunks, and metadata for retrieval	`knowledge-service`, `orchestration-layer`
`orchestration-layer`	coordinates services for query processing, retrieval, post-processing, completion, and answer checks	`pre-retrieval-processing`, `post-retrieval-processing`, `completion-service`, `evaluation-services`, `microsoft-qa-assist`, `copilot-for-azure`
`pre-retrieval-processing`	rewrites, expands, clarifies, or otherwise prepares queries before retrieval	`orchestration-layer`
`post-retrieval-processing`	reranks, expands context, filters, removes redundancy, and compresses retrieved chunks	`orchestration-layer`, `completion-service`
`completion-service`	uses processed query and chunks to formulate generated answers	`evaluation-services`
`evaluation-services`	evaluates retrieval quality, groundedness, relevance, harms, and post-completion quality	`feedback-assessment-pipeline`, `orchestration-layer`
`feedback-assessment-pipeline`	uses consented feedback and root-cause analysis to improve the inference pipeline and documentation	`evaluation-services`, `knowledge-service`
`microsoft-qa-assist`	customer-facing Q&A feature consuming Ask Learn outputs	`orchestration-layer`
`copilot-for-azure`	uses Ask Learn as grounding data and fallback self-service help inside Azure portal workflows	`orchestration-layer`

Data flow

Microsoft Learn articles are chunked, embedded, and saved with vectors in a database by the Knowledge Service. User queries go through query processing, embedding search, retrieval, re-ranking/context expansion/filtering/compression, completion, post-completion quality/safety checks, and response delivery. Feedback and consented chat histories feed a root-cause/assessment process that inspects the query, chunks, chunking strategy, and inference steps.

3 supporting quotes

“the Knowledge Service is responsible for breaking Microsoft Learn’s technical documentation into chunks, creating embedding vectors for each chunk, and saving the vectors and chunks in a database.”article, stated

“Other services consumed by the orchestration layer handle pre- and post-processing tasks, various evaluations to determine the accuracy and reliability of the answer”article, stated

“they perform root cause analysis on each response – regardless of whether the responses were positive or negative.”article, stated

Why this architecture

Ask Learn grew beyond naïve RAG because Copilot and Q&A needed reliable, relevant, accurate, and verifiable answers over a massive, changing documentation corpus. A service-oriented architecture let the team isolate chunking/vector access, pre/post processing, evaluation, and product integrations; golden datasets and feedback forensics addressed non-determinism and product-quality drift.

3 supporting quotes

“Naïve RAG gives you decent results. However, Microsoft’s customers expect reliable, relevant, accurate, and verifiable answers.”article, stated

“They designed a service-oriented architecture featuring a high-level orchestration layer responsible for calling individual services”article, stated

“The best place to start tackling the problem of non-determinism is by developing a “golden dataset.””article, stated

Models used: Azure OpenAI Embedding API · Azure OpenAI Completion API

Scale

The knowledge service must stay updated as hundreds of technical writers update hundreds of thousands of documents.
“when any one of hundreds of technical writers updates any one of hundreds of thousands of documents.”article, implied
The Azure portal supports hundreds of thousands, if not millions, of distinct operations.
“Given that the Azure portal supports hundreds of thousands – if not millions – of distinct operations”article, stated
Feedback root-cause analysis can take up to 30 minutes per response.
“requiring the team to spend up to 30 minutes for each response investigating how the query produced the results”article, stated
Q&A Assist was introduced in May 2023 after a few intense months of development.
“After just a few short, intense months of development, in May 2023, the team introduced Microsoft Q&A Assist”article, stated

Alternatives considered (roads not taken)

Naïve RAG with only query embedding, vector search, and completion

Rejected because: Microsoft judged naïve RAG insufficient for reliable, relevant, accurate, verifiable product answers.

“Naïve RAG gives you decent results. However, Microsoft’s customers expect reliable, relevant, accurate, and verifiable answers.”article, stated

One monolithic RAG service

Rejected because: The team used service-oriented orchestration so each major responsibility could be handled by individual services.

“They designed a service-oriented architecture featuring a high-level orchestration layer responsible for calling individual services”article, stated

Treat Ask Learn only as a direct Q&A answerer

Rejected because: Copilot also uses Ask Learn as grounding data for other handlers and as fallback self-service help.

“Ask Learn provides an anchor of reliable facts and context that other components use to inform their actions.”article, stated

“Second, Ask Learn serves as a fallback for the Copilot.”article, stated

Rely only on thumbs-up/down aggregate feedback

Rejected because: The team moved toward consented verbatim feedback and per-response root-cause analysis to diagnose actual failure modes.

“At first, the responses were aggregated as quality metrics, which helped the team find broad issues.”article, stated

“added a feature asking for the customers’ consent to share their chat history when giving feedback, along with verbatim feedback.”article, stated

Patterns

`rag-retrieval` (retrieval)

Ask Learn retrieves Microsoft Learn chunks as ground truth for Q&A and Copilot for Azure.

“One of the most ambitious and impactful projects our engineers have built recently is Ask Learn, an API that provides generative AI capabilities to Microsoft Q&A and the ground truth necessary to power the new Microsoft Copilot for Azure experience.”article, stated

`query-rewriting` (retrieval)

The advanced RAG pipeline rewrites, expands, and clarifies queries before retrieval.

“Pre-retrieval processing includes query rewriting, query expansion, and query clarification to better understand user intent.”summary, stated

`retrieval-ranking` (retrieval)

After retrieval, chunks are reranked, expanded, filtered, and compressed before response generation.

“Post-retrieval processing encompasses re-ranking retrieved document chunks, expanding chunks to provide additional context, filtering irrelevant or low-quality information, and compressing results into more compact forms.”summary, stated

Gotchas (what bit them)

Naive RAG was insufficient for Microsoft's production quality bar.

Root cause: Simple chunk/embed/retrieve/generate pipelines did not meet reliability, relevance, accuracy, and verifiability requirements.

Resolution: The team added advanced pre-processing and post-processing around retrieval and generation.

“While this provided "decent results," Microsoft's quality bar demanded reliability, relevance, accuracy, and verifiability.”summary, stated

Evaluation

The summary states that comprehensive evaluation frameworks and production monitoring are part of the LLMOps practice, but the available excerpt does not detail specific metrics.

“The resulting system demonstrates sophisticated LLMOps practices including advanced RAG architectures, continuous content pipeline management, comprehensive evaluation frameworks, and production monitoring at scale.”summary, stated

Outcomes

Ask Learn powers Microsoft Q&A generative AI capabilities and provides ground truth for Copilot for Azure.

The Microsoft post describes the system as operating at massive scale.

“This blog post details the story of collaboration between different product and engineering teams working together across Microsoft to build one of the world’s first retrieval-augmented generation (RAG) systems at such a massive scale.”article, stated