Gen AI On-Call Copilot for Engineering Support

uber · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: engineering on-call support

Uber needed to reduce repeated Slack support questions and long waits for on-call engineers by giving internal users a Genie copilot that could answer platform-engineering questions from existing internal documentation while respecting data-source access and limiting hallucinations.

Constraints:

• Uber teams received around 45,000 Slack-channel questions each month
• information was fragmented across Engwiki, internal Stack Overflow, requirement documents, and other sources
• many internal data sources could not be safely exposed in Slack channels
• answers needed to be grounded enough to reduce hallucinations and cite source URLs
• the system needed feedback, custom evaluation, document evaluation, and cost tracking for production operation

Architecture

Genie is a Slack-facing RAG copilot backed by an offline Spark ingestion pipeline and an online Knowledge Service. Spark jobs scrape approved internal knowledge sources, chunk content with LangChain, generate OpenAI embeddings through PySpark UDFs, and push vector artifacts through Sia and Terrablob. At serving time, Slack questions are embedded, Knowledge Service retrieves relevant chunks from the vector database, the retrieved sub-contexts and source URLs are passed to an LLM through Michelangelo Gateway, and user feedback plus context metadata flow through Kafka and Hive into dashboards and custom LLM-as-judge evaluation reports.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  internal-knowledge-sources["Internal knowledge sources"] --> spark-ingestion-pipeline["Spark ingestion pipeline"]
  spark-ingestion-pipeline --> embedding-generation["Embedding generation"]
  embedding-generation --> sia-vector-database["Sia vector database"]
  embedding-generation --> terrablob["Terrablob"]
  terrablob --> sia-vector-database
  slack-client["Slack client"] --> knowledge-service["Knowledge Service"]
  knowledge-service --> sia-vector-database
  knowledge-service --> michelangelo-gateway["Michelangelo Gateway"]
  michelangelo-gateway --> llm["LLM"]
  slack-client --> feedback-pipeline["Feedback pipeline"]
  feedback-pipeline --> evaluator["Evaluator"]
  evaluator --> llm
  internal-knowledge-sources --> document-evaluation-app["Document evaluation app"]
  knowledge-service --> cost-tracking["Cost tracking"]

Components

Data flow

Approved internal documentation is scraped into Spark dataframes, chunked, embedded with an OpenAI embedding model, and pushed to Sia/Terrablob for retrieval. When a user asks a question in Slack, Knowledge Service embeds the query, retrieves relevant chunks from the vector database, sends those chunks as cited sub-contexts to the LLM through Michelangelo Gateway, returns the answer with Slack action buttons, and streams feedback plus metadata through Kafka into Hive dashboards and evaluation pipelines.

Why this architecture

Uber chose RAG over fine-tuning because it could launch without curated training examples and could keep answers tied to current internal documentation. The surrounding Spark, vector database, feedback, evaluation, and source-citation machinery addresses the production risks of hallucination, stale documents, access control, cost attribution, and continuous tuning.

Models used: Genie · RAG · OpenAI embedding model · LLM · LLM as a Judge · Apache Spark · PySpark UDFs · LangChain · Sia · Terrablob · Knowledge Service · Michelangelo Gateway · Kafka · Hive · Slack

Scale

• Uber support channels received around 45,000 questions each month.
  “People ask around 45,000 questions on these channels each month, as shown in Figure 1.”article, stated
• Repeated support demand occurred across hundreds of Slack channels.
  “users often ask the same questions repeatedly, leading to a high demand for on-call support across hundreds of Slack channels.”article, stated
• The pusher architecture uses one bootstrap job and two Spark jobs to build and merge indexes.
  “A bootstrap job is triggered to ingest data from a data source to Sia, Uber’s in-house vector database solution. Then, two Spark jobs are triggered for the index build and merge”article, stated
• Since September 2023, Genie expanded to 154 Slack channels.
  “Since its launch in September of 2023, Genie has expanded its presence to 154 Slack channels”article, stated
• Genie answered over 70,000 questions.
  “and has answered over 70,000 questions.”article, stated
• Uber reports a 48.9% helpfulness rate for Genie.
  “Genie boasts a 48.9% helpfulness rate, showcasing its growing effectiveness.”article, stated
• Uber estimates Genie saved 13,000 engineering hours.
  “We estimate it’s saved us 13,000 engineering hours so far since its launch.”article, stated

Alternatives considered (roads not taken)

Fine-tune an LLM for on-call support

Rejected because: Fine-tuning needed curated high-quality diverse examples and ongoing compute to keep the model updated, while RAG could launch faster from existing documents.

Expose broad internal data sources directly to Slack and OpenAI

Rejected because: Uber limited ingestion to curated sources that were widely available to most engineers to avoid leaking sensitive data through embeddings or Slack answers.

Return answers without source-aware sub-contexts

Rejected because: Uber changed prompt construction to provide sub-contexts and source URLs so the LLM would answer from supplied evidence and cite source URLs.

Rely only on user answer feedback

Rejected because: Uber added custom evaluations, LLM-as-judge reports, and document-quality evaluation to tune retrieval, generation, and the knowledge base itself.

Force every unresolved question through another manual wait cycle

Rejected because: The interaction mode gives users next-step buttons for follow-up, resolution, or human escalation, reducing back-and-forth latency while preserving escalation when Genie cannot help.

Patterns

rag-retrieval (retrieval)

Genie uses a standard RAG path: internal documents are chunked and embedded, Slack questions are embedded at query time, relevant chunks are retrieved, and the retrieved context becomes the prompt for the LLM answer.

eval-driven-development (evaluation)

Uber built user feedback, custom hallucination/relevancy evaluations, LLM-as-judge reports, and document-quality evaluation into the Genie operating loop.

llm-as-judge (evaluation)

Genie evaluation pipelines use LLM-as-judge prompts to score both responses and source-document quality, then publish the metrics and explanations to reports.

human-in-the-loop-gate (support workflow)

Genie leaves an explicit path back to on-call engineers when the copilot cannot answer, and the feedback options distinguish resolved, partially helpful, wrong, and not-relevant cases.

privacy-preserving-processing (data governance)

Uber limited embeddings to pre-curated broadly accessible data sources and scoped embeddings to Slack channels to avoid leaking sensitive knowledge sources.

observability-tracing (operations)

Genie propagates UUIDs through Knowledge Service and Michelangelo Gateway into audit logs for cost tracking, and streams Slack feedback metadata through Kafka and Hive into dashboards.

Tradeoffs

Initial model strategy

Chose RAG over fine-tuning over fine-tuning an LLM on curated examples.

Rationale: RAG let Uber launch faster from existing internal documents instead of collecting diverse high-quality fine-tuning data and repeatedly retraining.

Knowledge-source scope

Chose pre-curated broadly accessible sources over embedding every potentially useful internal source.

Rationale: Restricting sources reduced leakage risk at the cost of excluding some information that might have helped answer edge cases.

Hallucination mitigation

Chose retrieved sub-contexts with source URLs in the prompt over letting the LLM answer from general model knowledge.

Rationale: The cited sub-context approach anchors answers to retrieved internal chunks and makes source provenance visible.

Operational feedback path

Chose Slack-native feedback plus separate evaluation pipelines over manual inspection of support answers after the fact.

Rationale: Immediate feedback and custom evaluations create structured data for tuning retrieval, generation, and document quality.

User interaction design

Chose action buttons for follow-up, resolution, and human support over a one-shot answer-only Slack bot.

Rationale: Buttons make it easier for users to continue the conversation, confirm resolution, or escalate when Genie is not the right responder.

Gotchas (what bit them)

RAG quality is capped by source-document quality.

Root cause: If internal documentation is unclear or inaccurate, the LLM cannot reliably produce high-quality answers no matter how strong the generation model is.

Resolution: Add document evaluation that scores source documents and returns actionable suggestions for improving the knowledge base.

Internal support knowledge is fragmented across multiple systems.

Root cause: Answers are spread across Engwiki, internal Stack Overflow, engineering requirement documents, and other locations, making direct search difficult.

Resolution: Scrape multiple approved sources into a common embedding and vector-search pipeline.

Hallucination risk persists unless retrieval context is constrained and cited.

Root cause: An LLM can generate misleading information if prompts do not force it to answer from relevant authoritative chunks.

Resolution: Retrieve relevant knowledge, add sub-contexts and source URLs to prompts, and require answers to cite the source URL.

Sensitive data sources cannot automatically be embedded or exposed in Slack.

Root cause: Some internal sources are not safe to expose to broad Slack audiences or to the embedding/generation path.

Resolution: Pre-curate only widely accessible sources and scope embeddings to appropriate Slack channels.

Answer feedback alone is not enough to tune retrieval and generation.

Root cause: Users may care about hallucination, answer relevance, or other use-case-specific metrics that require explicit evaluation workflows.

Resolution: Let users run custom evaluations and join context, responses, Slack metadata, and feedback into evaluation reports.

Support loops can still stall if users cannot easily continue, resolve, or escalate.

Root cause: The original on-call process involved several rounds of waiting for human responses and clarifications.

Resolution: Add interaction buttons for follow-up questions, resolution marking, and contacting human support.

Evaluation

Genie combines Slack feedback, dashboarded operational metrics, custom user evaluations for hallucination and answer relevance, an LLM-as-judge pipeline over context/responses/metadata, and a document-evaluation app that scores and recommends improvements to knowledge-base documents.

Metrics: Resolved · Helpful · Not Helpful · Not Relevant · hallucination · answer relevancy · document evaluation score · helpfulness rate · engineering hours saved · questions answered

Outcomes

• Genie expanded to 154 Slack channels after launch in September 2023.
• It answered more than 70,000 questions.
• Uber reports a 48.9% helpfulness rate.
• Uber estimates 13,000 engineering hours saved.
• The product added follow-up, resolution, and human-support actions to improve Slack support workflows.

This is a production internal support copilot at Uber scale: roughly 45,000 monthly Slack support questions motivated the project, the deployed system reached 154 Slack channels, answered over 70,000 questions, achieved a reported 48.9% helpfulness rate, and was estimated to save 13,000 engineering hours.