Error Handling in LLM Systems

uber · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: natural-language SQL generation for enterprise data access

Uber needed to let engineers, operations managers, and data scientists generate reliable SQL from natural-language questions despite huge internal schemas, sparse user prompts, hallucinated tables or columns, and high expectations that queries should run correctly.

Constraints:

Uber's data platform handles approximately 1.2 million interactive queries each month.
Operations contributes around 36% of interactive queries.
Large Tier 1 schemas can span more than 200 columns and consume 40-60K tokens.
Users may ask short, typo-filled, or underspecified questions.
Generated SQL must choose the right business domain, tables, columns, filters, and query style.

Architecture

Uber's QueryGPT evolved from a simple RAG few-shot prompt into a production multi-agent SQL-generation workflow. A user prompt is mapped to workspaces by an intent agent, relevant SQL samples and schemas are retrieved, a table agent lets users confirm or edit table choices, a column-prune agent removes irrelevant schema fields to avoid token blowups, and an LLM generates SQL. A standardized evaluation harness compares generated intent, datasets, executable SQL, output presence, and qualitative similarity to golden SQL.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  natural-language-query-interface["Natural-language query interface"] --> intent-agent["Intent agent"]
  intent-agent --> workspace-retrieval["Workspace retrieval"]
  workspace-retrieval --> table-agent["Table agent"]
  table-agent --> column-prune-agent["Column prune agent"]
  column-prune-agent --> sql-generation-llm["SQL generation LLM"]
  sql-generation-llm --> evaluation-harness["Evaluation harness"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Natural-language query interface`	Accepts English questions from analysts, engineers, operations, and support users.	`Intent agent`
`Intent agent`	Classifies the user question into business domains or workspaces.	`Workspace retrieval`
`Workspace retrieval`	Narrows retrieval to curated SQL samples and tables for domains such as Ads, Mobility, and Core Services.	`Table agent`
`Table agent`	Selects candidate tables and lets the user acknowledge or edit the table list.	`Column prune agent`
`Column prune agent`	Removes irrelevant columns from large schemas to control token use.	`SQL generation LLM`
`SQL generation LLM`	Generates SQL and an explanation from prompt, instructions, schemas, and samples.	`Evaluation harness`
`Evaluation harness`	Measures intent, table overlap, successful run, output presence, and qualitative SQL similarity.	—

Data flow

The user asks a natural-language data question. QueryGPT infers business intent and workspace, searches domain-specific samples and schemas, proposes tables for user acknowledgment or correction, prunes irrelevant columns from large schemas, wraps the selected context with Uber business instructions and the prompt, generates SQL, and evaluates the result against golden intent, datasets, SQL execution, output, and qualitative similarity.

“Every incoming prompt from the user now first runs through an “intent” agent. The purpose of this intent agent is to map the user’s question to one or more business domains/workspaces”article, stated

“we implemented a “Column Prune” agent, wherein we use an LLM call to prune the irrelevant columns from the schemas we provided to the LLM.”article, stated

Why this architecture

Uber decomposed QueryGPT into specialized agents because simple prompt-to-RAG retrieval degraded as tables and samples grew. Intent classification narrows the business domain, table selection brings humans into ambiguous choices, and column pruning keeps giant schemas inside token limits before the SQL-generation call.

“What we needed was an intermediate step, which classifies the user’s prompt into an “intent” that maps to relevant schemas and SQL samples.”article, stated

“The intent agent, table agent, and column prune agent each did an excellent job because they were asked to work on a single unit of work rather than a broad generalized task.”article, stated

Models used: LLMs · OpenAI GPT-4 Turbo · Vector databases · Similarity search · Intent agent · Table agent · Column Prune agent · LLM-based SQL similarity evaluator

Scale

Uber handles about 1.2 million interactive data-platform queries monthly.
“At Uber, our data platform handles approximately 1.2 million interactive queries each month.”article, stated
Operations contributes about 36% of the query volume.
“The Operations organization, one of the largest user cohorts, contributes to about 36% of these queries.”article, stated
Some schemas span more than 200 columns and can consume 40-60K tokens.
“some spanning over 200 columns. These large tables could use up as much as 40-60K tokens in the request object.”article, stated
The limited release averaged about 300 daily active users.
“With our limited release to some teams in Operations and Support, we are averaging about 300 daily active users”article, stated
Uber reported query authoring time falling from roughly 10 minutes to about 3 minutes.
“each query takes about 10 minutes to author, QueryGPT can automate this process and provide sufficiently reliable queries in about 3 minutes.”article, stated

Alternatives considered (roads not taken)

Simple few-shot RAG over SQL samples and schemas.

Rejected because: It worked on small sample sets but declined in accuracy as more schemas and SQL samples were onboarded.

“While this version of the algorithm worked well for a small set of schemas and SQL samples, as we started to onboard more tables and their associated SQL samples into the service, we started seeing declining accuracy”article, stated

Ask the LLM to consume full large table schemas.

Rejected because: Large schemas could exceed token limits, so QueryGPT needed a column-pruning step.

“Having 3 or more of these tables would break the LLM call since the largest available model (at the time) only supported 32K tokens.”article, stated

Let QueryGPT choose tables without user correction.

Rejected because: User feedback showed that table choices were sometimes wrong, so Uber added a table agent with ACK/edit flow.

“users who saw that the tables that were eventually picked by QueryGPT were not correct.”article, stated

Patterns

`multi-agent-orchestration` (SQL generation)

QueryGPT splits SQL generation into intent classification, table selection, column pruning, and final query generation.

“The intent agent, table agent, and column prune agent each did an excellent job because they were asked to work on a single unit of work”article, stated

`rag-retrieval` (schema and sample retrieval)

The system retrieves SQL samples and schemas relevant to the user's natural-language prompt and inferred workspace.

“vectorize it and do a similarity search (using k-nearest neighbor search) on the SQL samples and schemas to fetch 3 relevant tables and 7 relevant SQL samples.”article, stated

`human-in-the-loop-gate` (table selection)

The table agent asks users to approve or edit the candidate table list before query generation.

“The user would either select the “Looks Good” button or edit the existing list and modify the list of tables to be used by the LLM for query generation.”article, stated

`eval-driven-development` (SQL quality)

Uber uses a golden question-to-SQL evaluation set to track aggregate improvements, regressions, and error patterns.

“To track incremental improvements in QueryGPT’s performance, we needed a standardized evaluation procedure.”article, stated

`context-compaction` (token management)

The column-prune agent removes irrelevant schema columns before sending context to the LLM.

“we use an LLM call to prune the irrelevant columns from the schemas we provided to the LLM.”article, stated

Tradeoffs

Break broad SQL generation into specialized agents.

Chose Intent, table, and column-prune agents over One broad generalized LLM task.

Rationale: Small specialized tasks improved accuracy and made each step easier to evaluate or correct.

“asked to work on a single unit of work rather than a broad generalized task.”article, stated

Use human confirmation for table selection.

Chose User ACK/edit flow over Fully automatic table selection.

Rationale: Users reported wrong table choices, so the table step needed a correction gate.

“pick the right tables and send those out to the user to either “ACK” or edit the given list and set the right tables.”article, stated

Evaluate representative usage rather than every possible business question.

Chose Curated questions from logs over Attempting exhaustive coverage of Uber's dataset universe.

Rationale: Uber has too many datasets and business questions for the evaluation set to cover everything.

“Uber has hundreds of thousands of datasets with varying levels of documentation. Thus, it is impossible for the set of evaluation questions to fully cover the universe of business questions”article, stated

Gotchas (what bit them)

Similarity search over raw schemas and SQL samples lost accuracy as coverage grew.

Root cause: Natural-language prompts did not map cleanly to schema text or SQL snippets once more tables and samples entered the system.

Resolution: Uber introduced intent classification and workspaces to narrow the search radius.

“Doing a simple similarity search for a user’s natural language prompt (“Find the number of trips completed yesterday in Seattle”) on schema samples (CREATE TABLE…) and SQL queries”article, stated

Large schemas caused token-limit failures.

Root cause: Some Tier 1 tables had more than 200 columns, and multiple schemas could exceed available model context.

Resolution: The team added a Column Prune agent to remove irrelevant schema columns.

“were still seeing token limit issues because some requests included one or more tables that each consumed a large amount of tokens.”article, stated

Generated SQL can hallucinate nonexistent tables and columns.

Root cause: The LLM still sometimes invents schema elements even with retrieved context.

Resolution: Uber experimented with prompts, chat-style iteration, and a planned validation agent.

“we do see instances where the LLM would generate a query with tables that don’t exist or with columns that don’t exist on those tables.”article, stated

Evaluation can be noisy and non-deterministic.

Root cause: LLM outputs vary even when the QueryGPT service has not changed.

Resolution: Uber looks for longer-term error patterns rather than over-indexing on about 5% run-to-run metric changes.

“running the same evaluation with no changes to the underlying QueryGPT service can result in different outcomes.”article, stated

Evaluation

Uber curated golden question-to-SQL mappings from real logs and runs both vanilla and decoupled evaluations. It tracks intent accuracy, table overlap, successful execution, non-empty results, and LLM-scored qualitative SQL similarity against golden SQL.

Metrics: Intent accuracy · Table overlap score · Successful run · Run has output · Qualitative query similarity · Regression patterns over time

“We identified a set of real questions from the QueryGPT logs, and manually verified the correct intent, schemas required to answer the question, and the golden SQL.”article, stated

“For each question in the evaluation, we capture the following signals:”article, stated

Outcomes

QueryGPT reached limited release for Operations and Support teams.
Uber reported around 300 daily active users.
About 78% of surveyed users said generated queries reduced the time they would have spent writing SQL from scratch.
The architecture evolved through more than 20 algorithm iterations from hackathon prototype to production service.

The article gives concrete usage and productivity figures: 1.2 million monthly interactive queries, 36% from Operations, 10-minute-to-3-minute query authoring estimates, and 300 DAU in limited release.

“there were about 20+ iterations of the algorithm between the 2 detailed below”article, stated

“about 78% saying that the generated queries have reduced the amount of time they would’ve spent writing it from scratch.”article, stated