MLOps Platform Integration with Metaflow for Training and Serving Workflows

doordash · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Machine-learning platform orchestration

DoorDash needed a unified ML platform that lets ML engineers train, reproduce, and promote models across development and production-scale infrastructure without forcing every team to manage Kubernetes, Argo, namespaces, quotas, dependencies, and metadata directly.

Constraints:

ML development and production are not cleanly separated because a good development training run may be the artifact promoted to production.
Training can happen at production scale even while a project is nominally in development.
Reproducibility is hard because ML workflows are inherently non-deterministic.
Company-wide infrastructure must support over a dozen organizations with different budgets, resources, secrets, and namespaces.
Platform abstractions must remain extensible as ML techniques and compute providers change.

Architecture

DoorDash integrated Metaflow as the ML engineer-facing workflow layer over Kubernetes, Argo Workflows, blob/metadata storage, image-building decorators, and namespace-level resource management. Engineers write flows in a unified API, while the platform handles dependency images, metadata persistence, code persistence, execution across Kubernetes or workflows, and resource quotas through team namespaces and queues. This lets teams run production-scale development flows, reproduce artifacts, and promote successful runs without rerunning training.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ml-engineer-workflow[ML engineer workflow] --> metaflow-api[Metaflow API]
  metaflow-api --> image-builder[Image builder]
  metaflow-api --> metadata-store[Metadata store]
  metaflow-api --> argo-workflows[Argo Workflows]
  argo-workflows --> kubernetes-namespaces[Kubernetes namespaces]
  image-builder --> kubernetes-namespaces
  kubernetes-namespaces --> resource-queues[Resource queues]
  production-promotion[Production promotion] --> metaflow-api

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`ML engineer workflow`	User-authored Metaflow flow for feature engineering, training, and serving work.	`Metaflow API`
`Metaflow API`	Unified and extensible interface hiding compute and workflow details.	`Image builder`, `Metadata store`, `Argo Workflows`
`Image builder`	Custom decorator that creates and caches dependency images from base images and Python packages.	`Kubernetes namespaces`
`Metadata store`	Stores metadata needed to reproduce training runs and model artifacts.	`Metaflow API`
`Argo Workflows`	Workflow execution layer for launching ML steps.	`Kubernetes namespaces`
`Kubernetes namespaces`	Team-scoped compute, secret, and access boundaries.	`Resource queues`
`Resource queues`	Queue and quota layer for CPU, memory, and GPU resources across teams.	`Kubernetes namespaces`
`Production promotion`	Promotion of successful development-scale artifacts to production without rerunning training.	`Metaflow API`

Data flow

An ML engineer writes and runs a Metaflow training flow. Custom image decorators build and cache dependency images; metadata and code persistence record the run; Argo Workflows and Kubernetes execute the steps in the appropriate team namespace and queue; successful artifacts can then be promoted to production or used in serving workflows while retaining reproducibility metadata.

“we needed uh something to abstract the underlying compute infrastructure for our uh MLES.”transcript, stated

“we do depend on Argo workflows and uh Kubernetes. Uh we also have other compute providers. um that we leverage.”transcript, stated

Why this architecture

DoorDash chose Metaflow because ML practitioners needed a lower-cognitive-overhead API while still running on production infrastructure. The platform separates user-facing workflow authoring from compute/resource details and records enough metadata, dependencies, and code state to reproduce runs across users, times, namespaces, and execution environments.

“we want a unified and extensible experience. Uh we needed uh something to abstract the underlying compute infrastructure for our uh MLES.”transcript, stated

“they should more or less generate the same model with within the margin there, right?”transcript, stated

Models used: Metaflow · Argo Workflows · Kubernetes · Kueue

Scale

DoorDash supports over a dozen organizations with separate resource and budget needs.
“at Door Dash we have probably over a you know a dozen different orgs um that you know have their own uh uh you know budget and and resource requirements”transcript, stated
Team resource queues can include quotas such as 100 A10 GPUs and borrow unused capacity from other teams.
“team A has uh 100 uh A10 GPUs right and uh as their uh nominal quota right or resource quota”transcript, stated
The platform runs training at production scale even during development.
“you really just want to promote what was in development to to production, right? And that entails running at at production scale even while in dev.”transcript, stated

Alternatives considered (roads not taken)

Rerun a successful training workflow in production after development.

Rejected because: The team wants to promote the exact development run with good metrics instead of rerunning a non-deterministic workflow.

“if you run a training flow and you like the metrics it's not like you want to rerun that exact same training flow in production again for it to count.”transcript, stated

Expose raw compute infrastructure directly to ML engineers.

Rejected because: DoorDash wanted a unified lower-cognitive-overhead interface that abstracts compute while preserving extensibility.

“from the users of the platform right the ML practitioners the people are actually training the models and and deploying them into production um it's a it's a lower cognitive overhead”transcript, stated

Use one shared team namespace and quota model.

Rejected because: Different orgs have distinct budgets, secrets, access boundaries, and resource needs.

“different orgs um that you know have their own uh uh you know budget and and resource requirements uh as well as uh you know different secrets that they need to manage.”transcript, stated

Patterns

`durable-execution` (orchestration)

Metaflow persists metadata, dependencies, and code so ML workflows can be reproduced across environments and promoted safely.

“metadata persistence is also uh important for reproducing ML workflows right”transcript, stated

`structured-output-contract` (platform)

The platform exposes a public API and decorators that standardize images, metadata, and workflow execution while hiding underlying infrastructure.

“this basically takes the base image and then installs the the specified Python packages uh and mints a new image based off that and that image is cached.”transcript, stated

`batch-offline-processing` (orchestration)

Training flows execute through workflow infrastructure such as Argo and Kubernetes rather than a local notebook-only path.

“basically uh we do depend on Argo workflows and uh Kubernetes.”transcript, stated

Tradeoffs

Promote artifacts instead of rerunning training.

Chose Promoting the development run that produced good metrics. over Rerunning the same training flow in production..

Rationale: The good run may already have executed at production scale and rerunning introduces waste and non-determinism.

“you really just want to promote what was in development to to production, right? And that entails running at at production scale even while in dev.”transcript, stated

Abstract compute through Metaflow while preserving extensibility.

Chose A unified API over Kubernetes, Argo, blob storage, and other providers. over Forcing practitioners to manage each backend directly..

Rationale: The API lowers cognitive overhead while allowing multiple compute providers to coexist under the platform.

“You could think of this for um like you know running Spark jobs, you might use data bricks, you might use EMR.”transcript, stated

Namespace-scoped resource management.

Chose Per-team namespaces and queues with borrowable capacity. over A single undifferentiated resource pool..

Rationale: Teams need different quotas, secrets, access controls, and budgets, but spare resources should still be usable.

“these cues could actually borrow from each other. So let's say you know team A has uh 100 uh A10 GPUs”transcript, stated

Gotchas (what bit them)

ML production boundaries are blurry.

Root cause: A development training run can produce the exact artifact teams want to promote, and rerunning may waste compute or change results.

Resolution: DoorDash runs development flows at production scale and promotes successful artifacts while preserving run metadata.

“One thing to note about um uh ML projects is the line between production and development is uh you know uh very nuanced”transcript, stated

Reproducibility spans dependencies, metadata, code, users, and execution environments.

Root cause: ML workflows have inherent non-determinism and can run across different namespaces or compute paths.

Resolution: The platform handles dependency images, metadata persistence, and code persistence through Metaflow integration.

“there's additional complexity just because of the inherent non-determinism”transcript, stated

Company-wide ML infrastructure needs per-team isolation and sharing.

Root cause: Teams have different quotas, secrets, and access controls but may need to borrow unused capacity.

Resolution: DoorDash uses team namespaces and resource queues that can borrow spare resources.

“each team uh within Door Dash has their own uh namespace. uh each namespace has its own uh queue with a designated amount of resources”transcript, stated

Evaluation

The talk evaluates the platform through reproducibility expectations, organizational scalability, and the ability to run comparable training flows across users, namespaces, and execution environments.

Metrics: Over a dozen DoorDash orgs · Team-level queues and quotas including GPU, memory, and CPU · Comparable results across runs, users, namespaces, Kubernetes, and workflows

“if I run um a train workflow end to end, right? Um and then a week later Sunza or Salin run the same workflow, they should more or less generate the same model”transcript, stated

“including like you know for different GPU types, overall memory, CPU etc. Uh and these cues could actually borrow from each other.”transcript, stated

Outcomes

Gave ML engineers a unified workflow interface over DoorDash compute infrastructure.
Supported development-to-production promotion without rerunning successful training flows.
Scaled team resource management through namespaces, queues, and borrowable quotas.
Improved reproducibility through dependency images, metadata persistence, and code persistence.

The platform is company-wide, supports over a dozen orgs, per-team namespaces, and resource queues with GPU, memory, and CPU controls.

“supports self-service deployment workflows that significantly improve the ML engineer experience by eliminating the need to interact with multiple disparate systems.”summary, stated

“The platform team provides base templates that individual teams can override for their specific needs, creating a self-service model that reduces platform team toil.”summary, stated