Building a High-Quality Q&A Assistant for Database Research

airtable · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Q&A over complex Airtable bases

Airtable needed Omni to answer questions over real customer bases with large, noisy schemas, vague questions, and heterogeneous records without hallucinating, jumping to conclusions, or exceeding context limits.

Constraints:

large bases can have hundreds or thousands of records and schemas that exceed context windows
real schemas include deprecated, empty, similar, or ambiguous fields
LLMs can reason unpredictably, compound early mistakes, and hallucinate
users need citations so derived answers can be verified
the assistant needs evaluation coverage for common use cases and failure points

Architecture

Airtable built Omni as an agentic database-research assistant. It first explores high-level and view-specific schema context, then plans and replans as it discovers new data. It uses filtering, keyword search, semantic search, and fallback wider-scope searches to find records. Answers include inline citation tags pointing back to database or internet sources. A curated eval suite plus production usage examples supports iteration and model comparisons.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-question[User question] --> omni-agent[Omni agent]
  omni-agent --> schema-exploration-tool[Schema exploration tool]
  schema-exploration-tool --> airtable-base-data[Airtable base data]
  omni-agent --> planning-module[Planning module]
  planning-module --> hybrid-search-tools[Hybrid search tools]
  hybrid-search-tools --> airtable-base-data
  airtable-base-data --> citation-renderer[Citation renderer]
  citation-renderer --> omni-agent
  evaluation-suite[Evaluation suite] --> omni-agent

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User question`	Natural-language request for insight from an Airtable base.	`Omni agent`
`Omni agent`	Q&A assistant that plans, explores schema, searches records, analyzes data, and produces cited answers.	`User question`, `Schema exploration tool`, `Planning module`, `Hybrid search tools`, `Citation renderer`
`Schema exploration tool`	Provides high-level schema and detailed schema for currently relevant information.	`Omni agent`, `Airtable base data`
`Planning module`	Creates and revises research plans when newly discovered data changes the path.	`Omni agent`, `Hybrid search tools`
`Hybrid search tools`	Filter, keyword, and semantic search tools over base data with wider-scope fallback.	`Omni agent`, `Airtable base data`
`Airtable base data`	Customer records, schemas, tables, fields, and views being researched.	`Schema exploration tool`, `Hybrid search tools`, `Citation renderer`
`Citation renderer`	Inline citation tag system that keeps answers verifiable and compact.	`Omni agent`, `Airtable base data`
`Evaluation suite`	Curated and production-derived questions scored by deterministic or LLM-as-judge metrics.	`Omni agent`

Data flow

A user asks Omni a question about an Airtable base. The agent explores relevant schema context, forms a plan, filters candidate sources, combines keyword and semantic search, widens scope when no meaningful data is found, analyzes results, and answers with compact inline source citations. Eval suites and production feedback then update the test set for future iterations.

3 supporting quotes

“it mimics a human when trying to find answers from a database—explores the structure of each table, applies relevant filters, re-evaluates upon new information”article, stated

“We provide the LLM with tools to filter and search for base data. The filtering step narrows the search scope, while keyword and semantic search help identify results that are relevant for the user’s question.”article, stated

“The LLM cites the sources alongside each piece of information with inline citation tags. In the above example, the generation ends with “… a question before Justin ”.”article, stated

Why this architecture

Airtable split database Q&A into schema exploration, planning, retrieval, citation, and evaluation because simply dumping schema and records into an LLM is context-expensive and unreliable. Hybrid search handles both exact named entities and vague wording, replanning catches early mistakes, and citations reduce hallucination risk.

3 supporting quotes

“We started by providing the schema for all tables in the base, but even that could be too much for the LLM to process. For a larger base, the schema alone would consume most of the context window. Hence we break down data exploration into 2 steps:”article, stated

“By combining both search methods, we can prioritize exact matches for named entities while still finding relevant results even when search queries are vague”article, stated

“Citation of sources is important for LLM generated answers because citations allow users to verify information and reduce the risk of relying on potentially inaccurate responses.”article, stated

Models used: LLMs · LLM-as-a-judge scorers

Scale

Large bases can contain hundreds or thousands of records.
“The first and foremost challenge is: given hundreds and thousands of records that far exceeds LLM’s context window, how can we find an answer in a token-efficient manner?”article, stated
Airtable launched Omni in June 2025.
“In June 2025, Airtable launched Omni, an AI assistant that can build custom apps and extract insights from the vast amount of information stored in an Airtable base, such as customer feedback, marketing campaign data, and product details.”article, stated

Alternatives considered (roads not taken)

Provide all table schema to the LLM up front

Rejected because: For large bases, even schema alone can consume the context window.

“We started by providing the schema for all tables in the base, but even that could be too much for the LLM to process.”article, stated

Single-pass answering without replanning

Rejected because: Confusing schemas and multi-step tasks require backtracking when new data changes the path.

“In the system, we incorporate a planning step, as well as steps to replan upon discovery of new data. We built an evaluation system that captures complex scenarios where data schema is confusing and multiple explorations or backtracking may be required in the process.”article, stated

Keyword-only or semantic-only search

Rejected because: The system needs exact named-entity matches and fuzzy matching for vague or differently worded queries.

“tools to filter and search for base data. The filtering step narrows the search scope, while keyword and semantic search help identify results that are relevant for the user’s question. By combining both search methods, we can prioritize exact matches for named entities while still finding”article, stated

Patterns

`tool-use-orchestration` (orchestration)

Omni orchestrates schema exploration, filters, search, planning, analysis, and citation tools to answer database questions.

“We provide the LLM with tools to filter and search for base data.”article, stated

`hybrid-search` (retrieval)

Airtable combines keyword and semantic search to handle exact entity matches and vague user phrasing.

“By combining both search methods, we can prioritize exact matches for named entities while still finding relevant results”article, stated

`fallback-degradation` (reliability)

If no meaningful data is found, Omni searches again over a wider scope to recover from filtering mistakes.

“if no meaningful data was found, we perform the search again on a wider scope from the initial attempt.”article, stated

`eval-driven-development` (evaluation)

Airtable uses curated and production-derived eval questions with deterministic or LLM-as-judge scores to iterate on the system.

“The eval suite is a collection of questions curated from customer research and production usage.”article, stated

`llm-as-judge` (evaluation)

Some eval metrics are scored by LLM-as-a-judge scorers.

“We use deterministic or LLM-as-a-judge scorers to output result scores for various metrics we are interested in.”article, stated

Tradeoffs

Use contextual schema exploration

Chose high-level schema and focused detailed schema over full schema injection.

Rationale: Focused context keeps the model within token limits and reduces noise.

“we pare down the initial context to the most useful and salient information”article, stated

Use compact inline citations

Chose citation tags embedded alongside generated claims over large natural-language citation blocks.

Rationale: Inline tags are compact, model-agnostic, flexible, and later renderable as rich content.

“This style of citation follows the natural flow of conversation and has additional benefits of being compact, model agnostic and flexible.”article, stated

Use both deterministic and LLM judge evals

Chose mixed scoring for curated and production examples over manual-only quality review.

Rationale: The team needed to iterate quickly and compare regressions across models.

“they also enable us to compare performance across models and track regressions.”article, stated

Gotchas (what bit them)

LLMs can jump to wrong answers in complex schemas.

Root cause: Large table schemas, vague questions, and early reasoning mistakes compound.

Resolution: Use contextual schema exploration plus planning and replanning.

“they often jump to conclusions prematurely, compound initial mistakes, or hallucinate entirely inaccurate responses.”article, stated

Schema context can overwhelm the model.

Root cause: Large bases may have schema alone consume most of the available context window.

Resolution: Provide high-level schema and detailed schema only for actively viewed or relevant information.

“For a larger base, the schema alone would consume most of the context window.”article, stated

Filtering can accidentally omit the right table or column.

Root cause: The LLM can make random errors when narrowing data sources.

Resolution: Retry search over a wider scope when no meaningful data appears.

“As LLM is subject to random errors, it could overlook tables or columns in the filtering step.”article, stated

Evaluation

Airtable evaluates Omni with curated questions from customer research and production usage, deterministic and LLM-as-judge scorers, and examples selected for common use cases and failure points.

Metrics: curated eval questions · production-usage examples · deterministic scores · LLM-as-a-judge scores · regression tracking across models

“These examples are selected to represent Omni’s most common use cases and failure points.”article, stated

“We augment the list as more representative examples come in from usage.”article, stated

Outcomes

Airtable launched Omni as an assistant for building apps and extracting database insights
the described techniques support production-ready cited answers over complex bases
the next challenge is larger, more heterogeneous bases with low latency and longer iterative operations

The source does not provide adoption or accuracy-lift numbers; it frames success as production-readiness and reliability over large customer bases.

“deliver a production-ready assistant that users can trust.”article, stated

“our next challenge is scaling these capabilities to even larger, more heterogeneous bases with low latency”article, stated