Building a Resilient Embedding System for Semantic Search

airtable · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production embedding lifecycle for semantic search

Airtable needed semantic search over customer data while keeping embeddings consistent enough with MemApp, handling asynchronous generation, migrations, provider/model changes, data residency, sensitive embeddings, and failure recovery without putting huge vector payloads inside the single-writer primary database.

Constraints:

Architecture

Airtable chose eventual consistency and moved embedding generation/storage outside MemApp. MemApp owns embedding configs, data subscriptions, transaction-number-backed embedding states, and stale detection. When subscribed data changes, tasks generate embeddings asynchronously, the embedding service writes vectors to a separate vector database conditionally on transaction numbers, then confirms persisted state back to MemApp. For migrations, failures, provider changes, and residency moves, Airtable resets embedding configs: delete old vector data and recreate configs with new settings.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  base[Base] --> memapp[MemApp]
  memapp[MemApp] --> embedding-config[Embedding Config]
  embedding-config[Embedding Config] --> data-subscription[Data Subscription]
  data-subscription[Data Subscription] --> embedding-state[Embedding State]
  embedding-state[Embedding State] --> embedding-tasks[Embedding Tasks]
  embedding-tasks[Embedding Tasks] --> embedding-service[Embedding Service]
  embedding-service[Embedding Service] --> embedding-provider[Embedding Provider]
  embedding-service[Embedding Service] --> vector-database[Vector Database]
  vector-database[Vector Database] --> semantic-search[Semantic Search]
  vector-database[Vector Database] --> memapp[MemApp]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
MemApp Airtable's primary in-memory single-writer database for base reads/writes and embedding state tracking. Embedding Config, Embedding State, Embedding Tasks, Vector Database
Base A particular database instance whose data may need semantic search embeddings. MemApp
Embedding Config Abstraction mapping MemApp data to vector database storage, model strategy, storage configuration, and triggers. MemApp, Data Subscription, Vector Database
Data Subscription Declarative description of data the user wants embedded. Embedding Config, Embedding State
Embedding State Per-chunk transaction-state record tracking persisted and updated versions. MemApp, Data Subscription, Embedding Tasks
Embedding Tasks Asynchronous work created when subscribed data becomes stale or updated. Embedding State, Embedding Service
Embedding Service Generates embeddings with retry logic and writes them to the vector database. Embedding Tasks, Vector Database, Embedding Provider
Embedding Provider External or in-process model provider such as OpenAI or Bedrock-compatible models. Embedding Service
Vector Database External vector storage and search layer with transaction-conditional writes. MemApp, Embedding Config, Embedding Service, Semantic Search
Semantic Search User-facing search feature powered by embeddings. Vector Database

Data flow

A developer defines an embedding config in MemApp. MemApp provisions a vector table, creates embedding states for chunks, and updates lastUpdatedTransaction when data changes. Tasks generate embeddings asynchronously, the embedding service writes vectors to the vector database only if the transaction is newer than the stored one, MemApp records lastPersistedTransaction after the write, and semantic search uses the external vector database while stale states can be filtered or regenerated.

4 supporting quotes
“The solution? Embrace eventual consistency, and store and process embeddings outside of MemApp. This decision led us to design a data flow where embeddings are generated asynchronously and stored in a separate vector database — with their state tracked within MemApp.”article, stated
“Tasks are created to generate the embeddings for each config that saw data in its data subscription updated. Currently we do this within the transaction that the detection occurs.”article, stated
“Once embeddings are generated we store them in our vector database. We prevent out-of-order writes by making the insertion conditional on the transaction number for the write being greater than the transaction number stored in the vector DB.”article, stated
“We then confirm with MemApp that the write has happened, updating the lastPersistedTransaction. Once again we handle potential out of order writes by only ever increasing the value.”article, stated

Why this architecture

Airtable separated embedding work from MemApp because strong consistency would be too costly and slow. Transaction-number-backed embedding state gave the system enough ordering to detect stale data and prevent out-of-order writes, while embedding configs and resets made migrations, provider changes, and residency moves operationally simple.

3 supporting quotes
“Storing embeddings in MemApp would be prohibitively expensive because embeddings are often 10x the size of the underlying data.”article, stated
“2. Performance: Achieving strong consistency would require generating embeddings within transactions. This process would be too slow for bulk updates and would limit us to less capable, in-house models rather than leveraging top-tier providers like OpenAI.”article, stated
“Since MemApp is a serializable database, its transaction number is perfect for this — a BigInt that increments with each write. For each piece of data we embed we create an embedding state:”article, stated

Models used: OpenAI embedding models · Bedrock embedding models · AI embedding models

Scale

Alternatives considered (roads not taken)

Store embeddings inside MemApp for full consistency

Rejected because: Embedding vectors were too large for MemApp memory economics and generation inside transactions would be too slow.

“Our first fundamental design question was this: Should embeddings be stored within MemApp to ensure full consistency with the data? While appealing in theory, this approach had two major drawbacks:”article, stated
“Storing embeddings in MemApp would be prohibitively expensive because embeddings are often 10x the size of the underlying data.”article, stated

Maintain strong consistency through transaction-time generation

Rejected because: Transaction-time embedding generation would slow bulk updates and limit the team to less capable in-house models.

“This process would be too slow for bulk updates and would limit us to less capable, in-house models rather than leveraging top-tier providers like OpenAI.”article, stated

Incrementally patch every migration or invalidation case

Rejected because: A reset-based config lifecycle handled vector DB migrations, provider changes, model deprecation, cloning, corruption, and detection bugs with one operation.

“I realized we could handle all of these cases if any time the system detects that embeddings are, or about to become, invalid, we reset the embedding config.”article, stated

Patterns

batch-offline-processing (data-processing)

Embedding generation is handled asynchronously through tasks instead of synchronously inside MemApp transactions.

“Tasks are created to generate the embeddings for each config that saw data in its data subscription updated.”article, stated

fallback-degradation (resilience)

The reset operation provides a broad recovery and migration fallback when embeddings become invalid.

“if any time the system detects that embeddings are, or about to become, invalid, we reset the embedding config.”article, stated

data-quality-loop (consistency)

Embedding state tracks stale and persisted transaction numbers so stale results can be filtered or regenerated.

“We need to be able to determine which data has been embedded and what may be out of date (for filtering stale results or fixing up the vector database).”article, stated

privacy-preserving-processing (security)

Airtable treats embeddings as sensitive data and resets/recreates configs during residency moves to ensure old-region data is removed.

“Since embeddings are sensitive they must be moved as well.”article, stated
“our system then automatically deletes all existing embedding data as part of the normal cleanup process.”article, stated

Tradeoffs

Use eventual consistency for embeddings

Chose External asynchronous embedding generation and vector storage over Full consistency by storing embeddings in MemApp.

Rationale: Eventual consistency avoided prohibitive memory cost and transaction-time generation latency while preserving enough state for repair.

“The solution? Embrace eventual consistency, and store and process embeddings outside of MemApp.”article, stated
“This decision led us to design a data flow where embeddings are generated asynchronously and stored in a separate vector database”article, stated

Reset embedding configs for invalidation and migration

Chose Delete and recreate embedding configs over Case-specific migration and repair logic.

Rationale: The reset path handles many migration/failure cases with one operational pattern, at the cost of regeneration and temporary search unavailability.

“Deleting the old embedding config (cleaning up old data if it exists)”article, stated
“Resetting embeddings causes temporary unavailability of semantic search features.”article, stated

Pay regeneration cost during resets

Chose Regenerate embeddings when configs reset over Incremental updates that preserve availability but add storage/indexing complexity.

Rationale: Regeneration costs were manageable compared with storage and indexing costs, and p99.9 re-embedding under 2 minutes made downtime acceptable.

“Regenerating embeddings incurs expenses, but these are manageable compared to storage and indexing costs from incremental updates.”article, stated
“Given the rarity of resets and the speed of re-embedding (p99.9 under 2 minutes), this was acceptable.”article, stated

Gotchas (what bit them)

Strongly consistent embeddings would be too expensive and slow.

Root cause: Embeddings are much larger than source data and would have to be generated inside primary database transactions.

Resolution: Airtable accepted eventual consistency and stored/process embeddings outside MemApp.

“Storing embeddings in MemApp would be prohibitively expensive because embeddings are often 10x the size of the underlying data.”article, stated
“The solution? Embrace eventual consistency, and store and process embeddings outside of MemApp.”article, stated

Asynchronous vector writes can arrive out of order.

Root cause: Vector database updates and MemApp confirmations are separate operations with possible lag or lost writes.

Resolution: Airtable uses MemApp transaction numbers and only increases persisted state.

“Ordered data versions also lets us handle out-of-order write operations.”article, stated
“We prevent out-of-order writes by making the insertion conditional on the transaction number for the write being greater than the transaction number stored in the vector DB.”article, stated

Reset loops could create runaway costs and downtime.

Root cause: A broad reset mechanism can continuously invalidate and regenerate embeddings if not controlled.

Resolution: Airtable added metrics, alerts, rate limiting, and idempotent requests.

“We had to implement safeguards to prevent continuous reset loops, which could lead to spiraling costs and prolonged downtime.”article, stated
“We used a combination of metrics, alerts, rate limiting, and idempotent requests.”article, stated

Evaluation

The article does not describe an LLM evaluation harness; it describes operational correctness checks through transaction state, conditional vector writes, metrics, alerts, rate limiting, idempotent requests, and product-level handling of temporary search downtime.

Metrics: p99.9 re-embedding under 2 minutes · transaction-number ordering · metrics and alerts for reset loops · rate limiting for runaway resets

“Given the rarity of resets and the speed of re-embedding (p99.9 under 2 minutes), this was acceptable.”article, stated
“We used a combination of metrics, alerts, rate limiting, and idempotent requests.”article, stated

Outcomes

The system operates across Airtable bases, including large-scale generations users can trigger in the product and residency migrations such as US-to-EU moves.

“By accepting eventual consistency and designing for graceful handling of failures and migrations, we’ve built a foundation that supports powerful semantic search capabilities”article, stated
“Handling this kind of downtime is already necessary at the product level since users can trigger large scale generations in the product very easily”article, stated