Scaling Training Data Storage for Next-Generation Recommender Systems with User Sequences

meta · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI training data storage for recommender systems

Meta needed to store much longer user-sequence features for next-generation recommenders without letting duplicated training rows explode storage, power, latency, and GPU-training cost.

Constraints:

• User-sequence features made training rows roughly 100x larger than prior feature snapshots.
• Training data already operated at tens of exabytes across more than 10 data centers.
• The solution had to feed GPUs efficiently during training, not just reduce storage bytes at rest.
• A normalized design introduced joins that had to be optimized enough to beat the original single-table approach.

Architecture

Meta normalized training data by splitting lightweight training rows from a separate user-sequence table stored in ZippyDB. A special training-preprocessor connector joins rows and sequences at training time, while storage-server optimizations such as offloaded SSTable compaction, direct SSTable loading, RocksDB multi-scan, and schema-aware projection reduce write amplification, read latency, and over-fetching.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Training table"] --> D["Training preprocessor connector"]
  B["User sequence table"] --> C["ZippyDB sequence store"]
  C --> D
  D --> H["GPU training jobs"]
  E["Direct SSTable load pipeline"] --> C
  F["RocksDB multi-scan layer"] --> C
  G["Schema-aware projection layer"] --> C
  G --> H

Components

Data flow

Training rows remain in a warehouse-backed training table with lightweight references; user sequences move to a ZippyDB-backed indexable table keyed by sequence ID; the training preprocessor uses a special connector to join rows and sequences at training time; optimized SSTable loading, multi-scan reads, and schema-aware projection make the storage server efficient enough to feed GPU training jobs.

Why this architecture

The self-contained-row model made training-sample reconstruction simple but duplicated slowly-changing user features. Meta accepted a more complex join architecture because normalization could cut duplication, then optimized storage reads and preprocessing enough to make the split-table design cost-effective.

Scale

• User-sequence training data made rows about 100x larger.
  “Now the training growth was suddenly 100x larger.”transcript, stated
• Training data reached tens of exabytes across more than 10 data centers and 10% of Meta's power budget.
  “Now, training data is of tens exabytes stored in more than 10 data centers and cost the 10% of meta's power budget.”transcript, stated
• Optimizations made each storage server roughly 5x more efficient.
  “We ended up with a storage server that was roughly 5x more efficient than when we started.”transcript, stated
• Per-server read throughput rose from 200 MB/s to more than 1 GB/s.
  “support more than a gigabyte per second read throughput up from 200 megabytes per second when we started.”transcript, stated
• Supported user-sequence length grew from 2K events to 80K events.
  “scale the user sequence length from 2K events to 80K events, which is almost a 40x improvement”transcript, stated
• Sequence storage served 30+ production models.
  “Sequence storage is currently in production serving 30 plus models across various product groups.”transcript, stated

Alternatives considered (roads not taken)

Keep fully self-contained training rows

Rejected because: Self-contained rows were easy to reconstruct but duplicated user-sequence data heavily as sequences grew.

Store user sequences inline in every training row

Rejected because: Meta split user sequences into a separate table and kept lightweight references in the training table.

Use row-by-row insertion into ZippyDB

Rejected because: Row inserts triggered compaction and write-path overhead, so Meta built direct SSTable loading from external pipelines.

Execute sequence scans serially

Rejected because: Serial scans made latency the sum of all scans, so Meta added a native multi-scan API and parallel IO.

Fetch entire sequences and filter on the GPU side

Rejected because: Over-fetching increased network, CPU, and latency tax, so Meta filtered by schema, attributes, and time ranges earlier.

Patterns

batch-offline-processing (data-infrastructure)

Meta moved expensive compaction, optimization, indexing, and SSTable generation into external data pipelines before loading optimized files into serving storage.

Tradeoffs

Normalize training data into two tables.

Chose separate training rows and user-sequence rows over fully self-contained training rows.

Rationale: The split reduced duplication but required more complex on-the-fly reconstruction during training.

Move user sequences to ZippyDB.

Chose efficient indexable key-value storage over keeping sequence payloads inline in the warehouse-backed training table.

Rationale: The expected access pattern was point lookups by sequence ID.

Build storage-specific optimizations instead of abandoning the normalized design.

Chose SSTable direct load, multi-scan, and schema-aware projection over returning to the original single-table training data layout.

Rationale: The first normalized implementation was costly, but targeted optimizations made it practical and efficient.

Gotchas (what bit them)

Normalizing training data reduced duplication but made sample reconstruction harder.

Root cause: The training preprocessor now had to join training rows with separate user-sequence rows during training.

Resolution: Meta built a special connector and optimized the join and storage server path until the split-table design became cost-effective.

The first ZippyDB-backed version was not cheaper than the original design.

Root cause: ZippyDB had not been optimized for training workloads, and the preprocessor had more complex reconstruction work.

Resolution: Meta optimized both the training preprocessor side and storage server side.

Row-by-row insertion caused storage-server read amplification and write pressure.

Root cause: Frequent background compactions and multi-level log-structured tree traversal consumed server resources and flash IO.

Resolution: Meta generated optimized SSTables externally and loaded them directly into the storage server.

Serial scans caused GPU pipeline stall risk.

Root cause: RocksDB did not support native multiple scans, so ZippyDB scans were serial and total latency summed across scans.

Resolution: Meta implemented a native multi-scan API that coalesced adjacent IO and issued ranges in parallel.

Opaque key-values caused over-fetching.

Root cause: Models often needed only specific time ranges and attributes, but the old path fetched whole sequences and filtered on the GPU side.

Resolution: Meta made the query engine schema-aware and added feature projection and length truncation.

Evaluation

Meta evaluated the design through storage-server efficiency, read throughput, sequence length supported, data fetched, read IO count, and production adoption across models.

Metrics: storage server efficiency · read throughput per server · events per user sequence · data fetched compared with original approach · production models served · IO per read

Outcomes

• Storage-server efficiency improved roughly 5x.
• Per-server read throughput rose from 200 MB/s to more than 1 GB/s.
• Supported user-sequence length increased from 2K to 80K events.
• Sequence storage went into production for 30+ models across product groups.

The work addressed training data at tens-of-exabytes scale across more than 10 data centers and 10% of Meta's power budget.