Evolution of ML Model Deployment Infrastructure at Scale

faire · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: ML model deployment platform

Faire needed to move from monolith-based, PR-heavy ML deployments to a safer self-serve model deployment system that handled model versions, workloads, testing, observability, rollouts, and cleanup at scale.

Constraints:

• The old process required data scientists to put models in S3, change backend monolith code, register features, and wait for daily-at-best deploys.
• Different models had different traffic patterns, model sizes, scaling needs, library versions, and GPU requirements.
• Library upgrades and new architectures were constrained by a single Java-compatible monolith path.
• SageMaker and Comet added flexibility but also created a confusing 5+ step, 3-4 PR process.
• Production model deployment needed golden-data testing, shadow rollouts, metrics, monitors, rollback, and lineage.

Architecture

Faire built MMM, an internal UI and management layer over Comet, S3, SageMaker, GitHub, backend monolith integration, golden-data testing, metrics, monitors, gradual/expedited rollouts, and shadow tests. Data scientists register models and endpoint settings in one flow while MMM generates code, opens PRs, validates configuration, auto-merges after tests, updates Comet, deploys SageMaker endpoints, and tracks deployment state.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  comet-registry["Comet registry"] --> mmm-ui["MMM UI"]
  feature-store["Feature store"] --> mmm-ui
  mmm-ui --> github-code-generator["GitHub code generator"]
  github-code-generator --> validation-pipeline["Validation pipeline"]
  validation-pipeline --> sagemaker-deployment["SageMaker deployment"]
  sagemaker-deployment --> observability-layer["Observability layer"]
  observability-layer --> mmm-ui

Components

Data flow

A data scientist registers a model in Comet and features in the feature store, enters model and endpoint configuration in MMM, MMM creates deployable artifacts and GitHub changes, runs golden-data tests and validation, auto-merges when checks pass, updates Comet and production S3 buckets, deploys to SageMaker, and tracks rollout, shadow, abort, retire, metrics, and monitors.

Why this architecture

Faire kept SageMaker and Comet as battle-tested platform pieces but hid their operational complexity behind MMM because the flexible infrastructure had become too hard for data scientists to operate directly. The UI concentrated model configuration, validation, rollout, and observability in one self-serve flow.

Models used: Amazon SageMaker · Comet · Feature store · S3 · GitHub PRs · Golden-data tests · Shadow rollouts

Scale

• Faire's platform evolution covered about two years of model serving and deployment work.
  “you know about two years let's look at what some of these systems look like at fair at that time”transcript, stated
• The old manual process could take two to three days or more.
  “it's actually taking me two to three days to deploy if not more”transcript, stated
• MMM reduced deployment to about four hours of wall-clock time and roughly ten minutes of human effort.
  “we've gone from 3ish days to four hours for a deployment”transcript, stated
• The platform can track models deployed in multiple versions, setups, or regions.
  “maybe you've deployed the model in three different versions three different setups maybe in different regions”transcript, stated

Alternatives considered (roads not taken)

Embed models directly in the backend monolith.

Rejected because: One bottleneck could not handle diverse traffic, libraries, architectures, GPU needs, deploy speed, and versioning.

Expose SageMaker and Comet directly through YAML and PRs.

Rejected because: The system became a five-plus-step deployment flow with three to four PRs for data scientists.

Require human review after tests pass.

Rejected because: For the supported deployment flow, tests are strong enough that the PR can auto-merge after validation.

Patterns

canary-rollout (model deployment)

MMM supports gradual, expedited, and shadow rollouts before final production deployment.

observability-tracing (deployment state)

MMM adds metrics, monitors, endpoint status, deployment state, and model usage tracking.

eval-driven-development (golden-data checks)

Deployments run validation and golden-data tests before production rollout.

fallback-degradation (deployment safety)

Users can abort deployments and retire models through the managed flow.

Tradeoffs

Buy core platform primitives.

Chose Comet and SageMaker over Building or self-hosting every deployment primitive internally.

Rationale: The team used external tools for experiment tracking and real-time inference, then built a custom management layer where needed.

Generate deployment changes automatically.

Chose MMM code generation and auto-merge after tests over Manual PR authoring and review for routine deployments.

Rationale: Automation reduced human time while preserving validation and production safety.

Keep models as artifacts rather than repository-level releases.

Chose Versioned model artifacts over Repository release branches mapped to model versions.

Rationale: Artifacts preserve sharing and collaboration across models built on shared stacks.

Gotchas (what bit them)

Monolith deployment blocked ML platform maturity.

Root cause: Every model needed backend changes and shared library compatibility.

Resolution: Move model serving into SageMaker-managed endpoints with standardized containers and MMM orchestration.

External platform flexibility created user-facing complexity.

Root cause: SageMaker, Comet, batch jobs, backend code, metrics, and YAML configs had to be manually coordinated.

Resolution: Build MMM as a single UI and code-generation layer over those systems.

Model lineage disappears without explicit version tracking.

Root cause: Retrained models, packages, and online serving versions change independently.

Resolution: Use Comet and MMM to track artifacts, deployed versions, endpoint states, and rollback information.

Evaluation

Faire validates deployments with golden-data test files, configuration validation, shadow testing against production, endpoint health, deployment state tracking, and planned distributional-equivalence checks for shadow results.

Metrics: Golden-data test pass · Deployment wall-clock time · Human deployment time · Endpoint status · Shadow-test distributional equivalence · Rollback/version traceability · Monitors and metrics health

Outcomes

• Deployment complexity dropped from multiple PRs to a single MMM UI flow.
• Most deployments require zero lines of user-written deployment code.
• Deployment time dropped from about three days to about four hours with roughly ten minutes of human effort.
• Deployments gained testing, tracking, observability, shadow rollout, and cleanup support.

The platform reduced deployment from roughly three days to four hours, with most of the four hours being testing wall-clock time and human effort around ten minutes or less.