Enhanced Agentic RAG for On-Call Engineering Support

uber · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: On-call engineering support RAG

Uber needed Genie, its Slack-based on-call copilot, to answer engineering security and privacy questions accurately enough for broader deployment without misleading engineers or overloading subject matter experts.

Constraints:

The copilot must answer thousands of queries across multiple Slack help channels.
Security and privacy policies contain subtle distinctions by persona, geography, retention, classification, and sharing protocol.
Initial traditional RAG over PDF policy documents produced incomplete, inaccurate, or irrelevant answers.
Document loaders lost table structure and formatting needed for retrieval.
Manual SME evaluation was slow and could take weeks.

Architecture

Uber moved Genie from traditional RAG to Enhanced Agentic RAG. Offline document processing enriches Google Docs with custom loaders, table-aware chunking, LLM-powered markdown formatting, summaries, FAQs, keywords, embeddings, and feature-store artifacts. Near-real-time answer generation uses a Query Optimizer and Source Identifier before retrieval, hybrid vector plus BM25 retrieval, a Post-Processor Agent for de-duplication and context ordering, and an answer-generating LLM that returns cited responses in Slack. Batch evaluation uses LLM-as-Judge with SME answers and source context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  slack-user-query["Slack user query"] --> query-optimizer["Query Optimizer"]
  query-optimizer --> source-identifier["Source Identifier"]
  source-identifier --> hybrid-retrieval["Hybrid retrieval"]
  offline-document-enrichment["Offline document enrichment"] --> source-identifier
  offline-document-enrichment --> hybrid-retrieval
  hybrid-retrieval --> post-processor-agent["Post-Processor Agent"]
  post-processor-agent --> answer-generating-llm["Answer-generating LLM"]
  answer-generating-llm --> slack-interface["Slack interface"]
  slack-interface --> slack-user-query
  answer-generating-llm --> llm-as-judge-evaluator["LLM-as-Judge evaluator"]
  llm-as-judge-evaluator --> offline-document-enrichment

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Slack user query`	On-call engineer question from Slack help channels.	`Query Optimizer`
`Query Optimizer`	Refines ambiguous queries and decomposes complex questions into simpler auxiliary queries.	`Source Identifier`
`Source Identifier`	Narrows retrieval to the policy documents most likely to contain the answer.	`Hybrid retrieval`
`Offline document enrichment`	Loads Google Docs, preserves tables and bullets, uses LLM formatting, and writes summaries, FAQs, keywords, embeddings, and artifacts.	`Hybrid retrieval`, `Source Identifier`
`Hybrid retrieval`	Combines vector search with BM25 over enriched metadata.	`Post-Processor Agent`
`Post-Processor Agent`	De-duplicates retrieved chunks and orders context according to source-document positions.	`Answer-generating LLM`
`Answer-generating LLM`	Generates the final cited answer from original query, auxiliary queries, and post-processed context.	`Slack interface`, `LLM-as-Judge evaluator`
`Slack interface`	Returns the response to engineers in Slack help channels.	`Slack user query`
`LLM-as-Judge evaluator`	Scores batch responses against SME answers, instructions, and source context.	`Offline document enrichment`

Data flow

Documents are loaded, enriched, chunked, embedded, and stored with metadata and offline artifacts. At answer time, the user query is optimized, likely source documents are identified, vector and BM25 retrieval produce context, the post-processor de-duplicates and orders chunks, and the answer LLM writes a cited Slack response.

“Like any RAG architecture, EAg-RAG consists of two key components: offline document processing and near-real-time answer generation.”article, stated

“Finally, the original user query, optimized auxiliary queries, and post-processed retrieved context are passed to the answer-generating LLM”article, stated

Why this architecture

Traditional semantic RAG struggled with subtle policy distinctions and poorly formatted source documents. Uber added document enrichment, source-aware query optimization, hybrid retrieval, post-processing, and automated LLM-as-Judge evaluation so Genie could improve answer quality quickly without requiring weeks of SME assessment for every experiment.

“Simple semantic similarity can lead to retrieving irrelevant context, reducing the accuracy of the final answer.”article, stated

“The automated evaluation reduced experiment evaluation time from weeks to minutes, enabling faster iterations and more effective directional experimentation.”article, stated

Models used: LLM-powered Query Optimizer · LLM-powered Source Identifier · Post-Processor Agent · answer-generating LLM · LLM-as-Judge · LangChain · LangGraph · Michelangelo Genie

Scale

Genie supports thousands of queries across multiple Slack help channels.
“designed to provide real-time support for thousands of queries across multiple help channels in Slack®.”article, stated
SMEs curated a golden set of more than 100 test queries.
“SMEs curated a golden set of 100+ test queries based on their extensive experience handling domain engineers’ inquiries.”article, stated
The initial knowledge base included more than 40 engineering security and privacy policy documents.
“Uber’s repository of 40+ engineering security and privacy policy documents—stored as PDFs—and tested against the golden test set”article, stated
Policy tables could span more than five pages.
“many of our policy documents contain complex tables spanning more than five pages, including nested table cells.”article, stated
EAg-RAG improved acceptable answers by 27% relative and reduced incorrect advice by 60% relative.
“increasing the percentage of acceptable answers by a relative 27% and reducing incorrect advice by a relative 60%”article, stated
The LLM-as-Judge scores responses on a 0-5 scale.
“the LLM-as-Judge module scores responses on a 0-5 scale, with 5 being the highest quality.”article, stated

Alternatives considered (roads not taken)

Traditional RAG with vector retrieval over policy chunks.

Rejected because: Ambiguous queries and subtle policy distinctions led to irrelevant context and inaccurate answers.

“Traditionally, answer generation involves two steps: retrieving semantically relevant document chunks via vector search and passing them along with the user’s query and instructions, to an LLM.”article, stated

Continue using PDF policy documents and standard PDF loaders.

Rejected because: PDF loaders lost tables, bullets, and row/column context needed for reliable retrieval.

“existing PDF loaders often fail to correctly capture structured text and formatting (such as bullet points and tables).”article, stated

Use only vector search.

Rejected because: Uber added BM25 over enriched metadata and unioned it with vector retrieval for better relevance.

“we introduced an additional BM25-based retriever alongside traditional vector search.”article, stated

Rely on SME-only experiment evaluation.

Rejected because: Manual evaluation consumed significant SME bandwidth and took weeks.

“While Genie’s modular framework allowed easy experimentation, assessing improvements required significant SME bandwidth, often taking weeks.”article, stated

Patterns

`rag-retrieval` (retrieval)

Genie retrieves internal documentation and feeds it to an answer-generating LLM for cited Slack responses.

“supports the full RAG (Retrieval-Augmented Generation) pipeline, covering document loading, processing, vector storage, retrieval, and answer generation.”article, stated

`query-rewriting` (retrieval)

The Query Optimizer refines ambiguous queries and decomposes complex questions before retrieval.

“Query Optimizer refines the query when it lacks context or is ambiguous. It also breaks down complex queries into multiple simpler queries for better retrieval.”article, stated

`hybrid-search` (retrieval)

Vector search is combined with BM25 over enriched summaries, FAQs, and keywords.

“The final retrieval output is the union of results from the vector search and the BM25 retriever, which is then passed to the post-processing step.”article, stated

`multi-agent-orchestration` (orchestration)

Pre-processing and post-processing agents optimize queries, identify sources, and refine retrieved context around the answer LLM.

“we introduced LLM-powered agents in the pre-retrieval and post-processing steps to improve context relevance and enhance extracted content before answer generation.”article, stated

`llm-as-judge` (evaluation)

Uber uses an LLM-as-Judge module to score chatbot responses against SME responses and retrieved context.

“The LLM-as-Judge module evaluates chatbot responses using the user query, SME response, and evaluation instructions as context (C)”article, stated

Tradeoffs

Retrieval architecture

Chose Enhanced Agentic RAG with pre- and post-processing agents. over Traditional two-step vector RAG..

Rationale: Domain-specific policy questions needed optimized queries, source narrowing, and context refinement.

“Unlike traditional RAG, the agentic RAG approach allows seamless integration of different agents, making it easier to test and assess incremental improvements quickly.”article, stated

Document source format

Chose Google Docs with HTML formatting and custom loader. over PDFs and generic PDF loaders..

Rationale: Google Docs preserved structure better and provided access-control metadata for security-sensitive answers.

“we transitioned from PDFs to Google Docs, using HTML formatting for more accurate text extraction.”article, stated

Retrieval method

Chose Union of vector search and BM25 retrieval. over Semantic vector search alone..

Rationale: BM25 over enriched metadata improved relevance for precise policy-document chunks.

“This retriever fetches the most relevant document chunks using enriched metadata, which includes summaries, FAQs, and keywords for each chunk.”article, stated

Evaluation method

Chose LLM-as-Judge batch evaluation with SME responses. over Every experiment assessed manually by SMEs..

Rationale: Automation cut evaluation from weeks to minutes while still using SME-quality context.

“SMEs provide high-quality responses or feedback on chatbot-generated answers (SME responses).”article, stated

Gotchas (what bit them)

Standard PDF loaders broke policy-table semantics.

Root cause: Extracted text isolated table cells from row and column context, making chunking and retrieval unreliable.

Resolution: Move to Google Docs plus a custom loader, table-aware chunking, and LLM-powered markdown formatting.

“many table cells become isolated, stand-alone text, disconnecting them from their respective row and column contexts.”article, stated

Semantic similarity retrieved subtly wrong policy context.

Root cause: Security and privacy documents differ by persona, geography, retention, classification, and sharing rules.

Resolution: Use source identification, enriched metadata, BM25, and post-processing before generation.

“These distinctions can include variations in data retention policies, data classification, and sharing protocols across different personas and geographies.”article, stated

SME evaluation became an iteration bottleneck.

Root cause: Experiment evaluation required significant expert bandwidth and often took weeks.

Resolution: Automate batch evaluation with LLM-as-Judge while using SME responses as reference context.

“The automated evaluation reduced experiment evaluation time from weeks to minutes, enabling faster iterations and more effective directional experimentation.”article, stated

Early experiments plateaued.

Root cause: Many improvements yielded only marginal gains without a clear path to better answer quality.

Resolution: Adopt the configurable agentic RAG architecture to test incremental improvements.

“Marginal gains and plateauing accuracy. Many experiments yielded only slight accuracy improvements before plateauing, with no clear path for further enhancement.”article, stated

Evaluation

Uber used a 100+ query SME-curated golden test set, LLM-as-Judge batch evaluation with SME responses and source context, 0-5 scoring, correctness labels, reasoning, and feedback to compare RAG experiments.

Metrics: Acceptable answers relative improvement: 27% · Incorrect advice relative reduction: 60% · Golden test set: 100+ queries · LLM-as-Judge score: 0-5 · Experiment evaluation time: weeks to minutes

“In our use case, the LLM-as-Judge module scores responses on a 0-5 scale, with 5 being the highest quality.”article, stated

“increasing the percentage of acceptable answers by a relative 27% and reducing incorrect advice by a relative 60%”article, stated

Outcomes

Increased acceptable answers by a relative 27%.
Reduced incorrect advice by a relative 60%.
Reduced experiment evaluation time from weeks to minutes.
Enabled broader deployment across multiple security and privacy help channels.
Reduced support load for on-call engineers and SMEs.

Genie supports thousands of queries across Slack help channels, evaluated against 100+ SME-curated queries and 40+ security/privacy policy documents.

“With these improvements, the copilot bot can now scale across multiple security and privacy help channels to provide real-time responses to common user queries.”article, stated