Natural Language to SQL Query Generation at Scale

uber · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise natural-language SQL generation

Uber needed to reduce SQL authoring time across a very large data platform where users generated over a million interactive queries per month.

Constraints:

• Uber's platform handles 1.2 million interactive queries monthly.
• SQL authoring took about 10 minutes per query.
• Different domains needed domain-specific query generation.
• The system required evaluation across many iterations before limited production release.

Architecture

QueryGPT combines LLMs, vector databases, similarity search, domain workspaces, specialized intent/table/column pruning agents, and a comprehensive evaluation framework to generate SQL from natural language.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  querygpt-workspace["QueryGPT Workspace"]
  intent-agent["Intent Agent"]
  table-pruning-agent["Table Pruning Agent"]
  column-pruning-agent["Column Pruning Agent"]
  similarity-search["Similarity Search"]
  sql-generator["SQL Generator"]
  evaluation-framework["Evaluation Framework"]
  querygpt-workspace --> intent-agent
  querygpt-workspace --> similarity-search
  intent-agent --> table-pruning-agent
  table-pruning-agent --> column-pruning-agent
  column-pruning-agent --> sql-generator
  similarity-search --> sql-generator
  sql-generator --> evaluation-framework

Components

Data flow

A user prompt enters a workspace, specialized LLM agents infer intent and prune relevant tables and columns, vector similarity retrieves related context, the SQL is generated, and evaluation tracks quality across iterations.

Why this architecture

Uber decomposed query generation into domain workspaces and specialized pruning agents because platform scale and schema complexity made one generic prompt insufficient.

Models used: Large language models · Vector databases · Similarity search

Scale

• Uber handles 1.2 million interactive queries monthly.
  “handles 1.2 million interactive queries monthly.”summary, stated
• Query authoring time dropped from about 10 minutes to 3 minutes.
  “reducing query authoring time from approximately 10 minutes to 3 minutes.”summary, stated
• The system went through more than 20 iterations.
  “the system evolved through 20+ iterations into a production service”summary, stated
• The limited release reached 300 daily active users.
  “The limited release achieved 300 daily active users”summary, stated
• 78% of users reported significant time savings.
  “78% reporting significant time savings”summary, stated
• Operations contributes 36% of all queries.
  “Uber's Operations organization which contributes 36% of all queries.”summary, stated

Alternatives considered (roads not taken)

Manual SQL authoring

Rejected because: It took about 10 minutes per query and limited productivity at Uber's query volume.

Single monolithic LLM agent

Rejected because: Uber used specialized agents for intent, table pruning, and column pruning.

Patterns

multi-agent-orchestration (agent-architecture)

QueryGPT uses specialized intent, table-pruning, and column-pruning LLM agents.

query-rewriting (query-generation)

The system generates SQL from natural-language prompts.

rag-retrieval (retrieval)

Vector databases and similarity search provide retrieval context for SQL generation.

eval-driven-development (quality)

A comprehensive evaluation framework guided more than 20 production iterations.

Tradeoffs

Decompose query generation into specialized agents

Chose Intent, table-pruning, and column-pruning agents over One general SQL-generation step.

Rationale: Decomposition helps manage domain and schema complexity at Uber scale.

Use domain workspaces

Chose Workspace-scoped query generation over One global schema context.

Rationale: Domain-specific workspaces improve relevance for different business areas.

Gotchas (what bit them)

Enterprise schemas are too broad for naive SQL generation.

Root cause: Users need the right domain, tables, and columns before the SQL can be correct.

Resolution: Use workspaces plus intent, table-pruning, and column-pruning agents.

Hackathon prototypes need many iterations before platform-scale release.

Root cause: Query generation quality and domain fit must be validated against real user workflows.

Resolution: Run more than 20 iterations and a limited release before broader deployment.

Evaluation

Uber used a comprehensive evaluation framework and reported query-time reduction, daily active users, and user-reported time savings.

Metrics: Query authoring time · Daily active users · User-reported time savings · Iteration count

Outcomes

• Query authoring time fell from about 10 minutes to about 3 minutes.
• The limited release reached 300 daily active users.
• 78% of users reported significant time savings.

Uber handles 1.2 million interactive queries monthly; Operations contributes 36% of all queries.