Building a Production Data Agent for 90,000 Tables at Scale

openai · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data discovery and analytics agent at exabyte scale

OpenAI needed internal users to answer data questions across 90,000 datasets without spending hours discovering which tables were semantically correct, how they joined, or which caveats applied before writing SQL.

Constraints:

• the data platform stored 1.5 exabytes across 90,000 datasets and served about 4,000 internal users
• many tables looked similar but differed in grain, join semantics, logged-out-user inclusion, ownership, freshness, and caveats
• a bare schema was insufficient for the model to choose correct tables
• the agent had to support Slack, web, IDE, and Codex CLI/MCP entry points
• tool access needed to be curated because overlapping metadata tools confused the model
• institutional knowledge retrieval had to respect access controls

Architecture

OpenAI built a deliberately simple data agent around GPT-5.5, an agent runtime, a six-layer context assembly system, and a curated tool set. Offline pipelines merge table usage metadata, human annotations, and Codex-enriched pipeline-code descriptions into table descriptions that are embedded for retrieval. At runtime, a user question is embedded, matched against the table vector store with semantic and exact search, augmented with access-controlled institutional knowledge and memory, then sent through an agent loop that writes SQL, runs tools, observes results, corrects itself, and returns the answer, SQL, and referenced tables.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-entry-points[User entry points] --> agent-runtime[Agent runtime]
  agent-runtime --> gpt-5-5[GPT-5.5]
  agent-runtime --> context-assembly-layer[Context assembly layer]
  context-assembly-layer --> table-description-index[Table description index]
  table-description-index --> codex-enrichment-pipeline[Codex enrichment pipeline]
  context-assembly-layer --> institutional-knowledge-service[Institutional knowledge service]
  context-assembly-layer --> memory-store[Memory store]
  agent-runtime --> curated-tools[Curated tools]
  curated-tools --> warehouse-and-platform-systems[Warehouse and platform systems]
  context-assembly-layer --> warehouse-and-platform-systems

Components

Data flow

A user asks in Slack, the web portal, an IDE, or Codex CLI through MCP. The question is embedded with the same model used for offline table descriptions, the context layer retrieves matching tables plus institutional knowledge and memory, the runtime sends that context to GPT-5.5, and the tool loop lets the agent write SQL, execute it, inspect results, retry, and return a verified answer with reasoning, SQL, and table references.

Why this architecture

OpenAI kept the agent itself simple because the failure mode was not lack of agent complexity; it was lack of the right semantic context. A single strong model, a runtime harness, curated tools, and carefully engineered context assembly were easier to operate than routers, fine-tunes, and overlapping retrieval systems, while the unified data platform made table meaning legible enough for the model to reason.

Models used: GPT-5.5 · OpenAI embedding model · Codex · Codex Skills

Scale

• The platform stored 1.5 exabytes across 90,000 datasets as of May 2026.
  “OpenAI’s data platform stores 1.5 exabytes across 90,000 datasets and serves ~4,000 internal users as of May 2026.”article, stated
• The data agent served about 4,000 internal users through the data platform.
  “OpenAI’s data platform stores 1.5 exabytes across 90,000 datasets and serves ~4,000 internal users as of May 2026.”article, stated
• Codex enrichment runs nightly in batches of 100 to 200 tables with each table taking 5 to 10 minutes.
  “It runs in batches of 100 to 200 tables, with each table taking 5 to 10 minutes. By reading the code, it captures what a table actually contains, how it is derived, how fresh it is, and when to use it instead of a similar table.”article, stated
• The curated production tool surface is around 13 tools.
  “Tools. The agent has access to a small, curated set of 13 tools. These cover company context lookups, internal knowledge bases, big data systems like Airflow and Spark, and metadata services.”article, stated
• The same Codex investment supported migration of 10,000 DAGs, 90,000 tables, and 600 petabytes in two months.
  “And the same investment in Codex that powers the agent has let the team do things most companies consider impossible, like migrating thousands of DAGs, 90,000 tables and 600 petabytes between clouds in two months.”article, stated
• The support workflow amplified an engineer to around one hundred fixes per day.
  “The loop that used to take an engineer a few hours per ticket now lets that engineer dispatch around a hundred fixes per day.”article, stated

Alternatives considered (roads not taken)

Add routers, multiple LLMs, fine-tuning, and specialized retrieval stacks

Rejected because: Each extra moving part could add cost, latency, and failure modes, while OpenAI's unified data foundation let a simple single-model agent work.

Expose about 40 metadata/orchestration/big-data tools

Rejected because: The model chose wrong tools and got conflicting overlapping answers, so OpenAI reduced the surface to around 13 non-overlapping tools.

Embed all historical queries as context

Rejected because: Most company queries are exploratory one-offs, so OpenAI ranked query examples by trustworthiness instead of treating them equally.

Use detailed step-by-step prescriptive prompts

Rejected because: The agent followed the instructions and produced worse answers; high-level goals plus good context worked better.

Build a data agent before fixing the data foundation

Rejected because: If company data is scattered or inconsistent, no agent architecture can make it legible enough to answer reliably.

Patterns

rag-retrieval (context-assembly)

The agent retrieves table descriptions, institutional knowledge, memory, and runtime context before asking GPT-5.5 to answer.

tool-use-orchestration (agent-runtime)

The runtime turns GPT-5.5 into an action-capable agent by dispatching tool calls, feeding results back, and looping until the answer is correct.

agent-memory (context-assembly)

Corrections and learnings from previous conversations are stored globally or personally and applied to future retrieved context.

retrieval-ranking (retrieval-quality)

Historical queries are ranked by trustworthiness so canonical dashboard queries influence the model more than one-off exploratory SQL.

privacy-preserving-processing (access-control)

Institutional knowledge is retrieved through an access-controlled service so the agent does not expose documents the user cannot access.

Tradeoffs

Keep the agent architecture simple

Chose single GPT-5.5 model, runtime, context layer, and curated tools over routers, multiple models, fine-tuning, and special post-training.

Rationale: The team reduced cost, latency, and operational failure modes because strong context infrastructure carried reliability.

Invest in context assembly instead of agent cleverness

Chose six layers of curated context before model invocation over letting a bare schema and model reason alone.

Rationale: The model could not distinguish semantically similar tables from schema alone.

Use a curated 13-tool surface

Chose right non-overlapping tools over every available metadata/orchestration/big-data tool.

Rationale: Reducing overlap prevented the model from picking near-duplicate tools and conflicting answers.

Rank historical queries by trust

Chose dashboard and data-scientist-authored query examples ranked highest over embedding every historical query equally.

Rationale: The retrieval corpus needed to favor reusable correct patterns over exploratory one-offs.

Use high-level prompt guidance

Chose goal-oriented instructions with strong context and tools over detailed procedural prompting for each question type.

Rationale: Prescriptive prompting produced worse answers, while modern models planned better when given enough information.

Gotchas (what bit them)

SQL generation was not the hard part.

Root cause: At 90,000 datasets, users first had to find semantically correct tables, understand grain, joins, and caveats, and avoid lookalike tables.

Resolution: OpenAI invested in context assembly from metadata, human annotations, Codex enrichment, institutional knowledge, memory, and runtime context.

Too many tools made the agent worse.

Root cause: Overlapping metadata, orchestration, and big-data tools returned similar/conflicting information that the model struggled to choose among.

Resolution: The team capped the tool surface at around 13 tools and removed overlapping tools.

Historical query retrieval can teach bad habits.

Root cause: Most historical queries are exploratory one-offs rather than canonical examples of correct table usage.

Resolution: OpenAI ranked past queries by trustworthiness so heavily used dashboard queries had more influence.

Overly prescriptive prompts degraded answer quality.

Root cause: Detailed step-by-step instructions constrained planning even when modern models had enough context to reason.

Resolution: The team shifted to high-level guidance and relied on context plus tools.

Agent reliability depended on unglamorous data-platform foundations.

Root cause: A data agent's source of truth spans the whole company, not just a repository, so inconsistent data stores and conventions would make context illegible.

Resolution: OpenAI relied on a unified data lake, monorepo-produced tables, conventions, duplicate-column policing, and table annotations.

Evaluation

The article describes production validation through verified-answer behavior, SQL execution and correction loops, table/source transparency, context quality, tool-surface failures and fixes, query-ranking improvements, and operational outcomes from adjacent Codex workflows.

Metrics: answer returned with SQL and referenced tables · correctness after SQL execution and result inspection · tool confusion before and after reducing from around 40 to 13 tools · copying good query patterns after trust ranking · migration completion time · release-agent incident count · support fixes per engineer per day

Outcomes

• OpenAI's internal data agent reliably answered questions across the data platform with answer, SQL, tables, and reasoning attached.
• The agent worked with a simple architecture because OpenAI invested in unified data infrastructure and context assembly.
• Codex enrichment added table-level descriptions from pipeline code each night.
• The same Codex infrastructure helped migrate thousands of DAGs, 90,000 tables, and 600 petabytes between clouds in about two months.
• A release agent ran open-source patch releases for three to four months without human involvement or a single incident.
• The support loop amplified engineers from hours per ticket to around a hundred fixes per day.

OpenAI reports 1.5 exabytes, 90,000 datasets, about 4,000 data-platform users, 100-200-table nightly Codex batches, around 13 agent tools, 600 petabytes migrated, 5,500 support users, and about 100 support fixes per day per engineer.