Multimodal Feature Stores and Research-Engineering Collaboration

runway · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Generative AI training data infrastructure

Runway needed infrastructure for querying, precomputing, and streaming massive multimodal datasets to distributed generative model training jobs while keeping research and engineering workflows aligned.

Constraints:

Datasets include video, images, text, embeddings, depth maps, and metadata.
Videos are large, slow to decode, and impractical to copy wholesale to each training machine.
Distributed training fans out requests across many machines, GPUs, and data loader workers.
Research workflows need flexibility without becoming one-off scripts that engineering must rewrite.

Architecture

Runway's multimodal feature store centralizes semantic search, precomputed features, and batch retrieval for distributed training, while shared Python/Pydantic codebases and Docker-based tooling keep researchers close to production workflows.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  raw-multimodal-dataset["Raw Multimodal Dataset"]
  multimodal-feature-store["Multimodal Feature Store"]
  feature-precomputation["Feature Precomputation"]
  research-query-interface["Research Query Interface"]
  distributed-training-jobs["Distributed Training Jobs"]
  raw-multimodal-dataset --> multimodal-feature-store
  multimodal-feature-store --> feature-precomputation
  multimodal-feature-store --> distributed-training-jobs
  multimodal-feature-store --> research-query-interface
  feature-precomputation --> multimodal-feature-store
  research-query-interface --> multimodal-feature-store
  distributed-training-jobs --> multimodal-feature-store

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Raw Multimodal Dataset`	Stores videos, images, text, audio, and metadata.	`Multimodal Feature Store`
`Multimodal Feature Store`	Stores precomputed features and supports semantic search and batch retrieval.	`Feature Precomputation`, `Distributed Training Jobs`, `Research Query Interface`
`Feature Precomputation`	Computes embeddings, depth maps, and other model inputs ahead of training.	`Multimodal Feature Store`
`Research Query Interface`	Lets researchers locate semantically relevant training subsets.	`Multimodal Feature Store`
`Distributed Training Jobs`	Stream selected rows and columns into GPU training workers.	`Multimodal Feature Store`

Data flow

Raw multimodal assets and metadata enter the feature store, expensive features such as embeddings and depth maps are precomputed, researchers query semantic subsets, and distributed training jobs stream only the needed rows and columns.

“Their solution involves building what they call a "multimodal feature store" that enables semantic search over unstructured data, pre-computation of features like embeddings and depth maps, and efficient streaming of data batches to distributed training infrastructure.”summary, stated

“Instead, you need to stream batches of data, downloading only the portions relevant to the current training step on each machine.”summary, stated

Why this architecture

The feature store shifts complexity to training-data access, where multimodal generative workloads need semantic querying, expensive feature reuse, and high-throughput streaming rather than low-latency online feature serving.

“The multimodal feature store inverts these priorities. For generative AI workloads, inference is relatively simple—you're typically just passing a prompt, an image, or some combination of modalities.”summary, stated

“The complexity shifts entirely to the training side, where you need to manage massive datasets containing videos, images, audio, text, and various computed features derived from these raw inputs.”summary, stated

Models used: Diffusion models · CLIP · Midas

Scale

Distributed training spans many machines and GPUs with multiple data loader workers.
“Modern AI training jobs run on many machines, each with multiple GPUs, each GPU potentially having multiple data loader workers.”summary, stated
Video datasets are too large to download entirely to every worker.
“But for video and image data, this is completely intractable.”summary, stated

Alternatives considered (roads not taken)

Download the full dataset to every training machine

Rejected because: That approach becomes intractable for video and image data.

“For small datasets (like the text data Rosenthal worked with at Square), you can simply download the entire dataset to each machine. But for video and image data, this is completely intractable.”summary, stated

Researcher handoff scripts to production engineers

Rejected because: One-off handoffs prevent code reuse, testing, and conventional software engineering practices.

“The alternative—researchers handing off one-off scripts—means reinventing models from scratch each time and losing the benefits of conventional software engineering.”summary, stated

YAML-heavy cloud training abstractions

Rejected because: They force researchers away from normal development workflows.

“Many cloud training tools require writing custom YAML files, using domain-specific languages, or rewriting code to fit the tool's paradigm.”summary, stated

Patterns

`context-engineering` (data-context)

The feature store curates the training context available to researchers by making multimodal subsets and precomputed features searchable and reusable.

“Researchers might want to ensure semantic diversity in training data, find specific subsets for fine-tuning, or explore the dataset in ways that require understanding the content semantically rather than just matching metadata fields.”summary, implied

`eval-driven-development` (engineering-practice)

Runway emphasizes shared codebases, tests, CI/CD, Docker, and Python/Pydantic so research work can be tested and reused instead of handed off as isolated scripts.

“The approach also emphasizes shared codebases between researchers and engineers, using Python/Pydantic instead of YAML configurations, and creating tooling that abstracts away cloud infrastructure complexity while maintaining Docker-based workflows.”summary, stated

Tradeoffs

Precompute expensive multimodal features

Chose Store embeddings and depth maps in the feature store over Compute every feature on the fly during training.

Rationale: Precomputation lets researchers reuse expensive features and experiment with configurations without recomputing them each time.

“Pre-computation enables researchers to experiment with different configurations—for example, trying different embedding models beyond CLIP—without repeatedly computing expensive features.”summary, stated

Keep researchers in normal development workflows

Chose Python, Pydantic, Docker, and IDE-friendly tooling over Cloud-specific YAML and DSL workflows.

Rationale: Lower workflow friction helps researchers run cloud training while retaining software engineering practices.

“The ideal experience: write code in your preferred IDE on your laptop in a shared codebase, hit "run," and have it execute on cloud GPUs with low latency.”summary, stated

Gotchas (what bit them)

Traditional feature-store assumptions do not map cleanly to generative video training.

Root cause: Inference-serving parity is less central than training-side multimodal data management.

Resolution: Invert the feature-store focus toward semantic querying, precomputation, and streaming training batches.

“Traditional feature stores, as used in tabular machine learning for applications like fraud detection, focus heavily on low-latency inference serving and maintaining parity between training and inference features.”summary, stated

Research-to-production walls create duplicated model work.

Root cause: Researchers iterate in flexible environments while engineers later reimplement production versions.

Resolution: Use shared codebases with examples, tests, CI/CD, and Docker conventions.

“This prevents code reuse and makes it difficult to apply software engineering best practices.”summary, stated

Evaluation

The case is a technical discussion of infrastructure requirements and organizational workflow rather than a benchmarked system report.

Metrics: Training data retrieval practicality · Research workflow reuse · Distributed training data access

“The discussion centers on the unique infrastructure challenges of training and serving multimodal AI models at scale”summary, stated

Outcomes

The feature store pattern supports semantic search over unstructured multimodal training data.
Precomputed features can be reused across experiments.
Shared codebases reduce the handoff wall between researchers and engineers.

The design targets large distributed training jobs where many machines, GPUs, and data loader workers stream selected multimodal data.

“The feature store becomes the canonical place where embeddings and other computed features live, enabling downstream researchers to query and filter data for experiments.”summary, stated