State of Production Machine Learning and LLMOps in 2024

zalando · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production ML and LLMOps operating model

The Zalando talk frames production ML in 2024 as a lifecycle, architecture, monitoring, security, compliance, and team-design problem rather than a one-time model-building project, with LLMs accelerating existing MLOps pressures.

Constraints:

• Production ML systems need monitoring, security, compliance, and risk assessment as they mature.
• Machine-learning products keep changing after deployment through new model versions and capability extensions.
• Teams must add personas beyond data scientists as model count and production complexity increase.
• Deployment timing should be governed by proportionate risk, human impact, and domain-expert alignment.

Architecture

The talk presents a production-ML operating model: move from projects to products, add cross-functional squads around ML systems, scale automation/security/observability with MLOps maturity, route artifacts through approval and production monitoring, and calibrate deployment overhead to risk, compliance, and user impact.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business_project_delivery["Business project delivery"] --> ai_product_lifecycle["AI product lifecycle"]
  ai_product_lifecycle --> cross_functional_ml_squad["Cross-functional ML squad"]
  ai_product_lifecycle --> mlops_infrastructure["MLOps infrastructure"]
  cross_functional_ml_squad --> mlops_infrastructure
  mlops_infrastructure --> ops_approval_gate["Ops approval gate"]
  ops_approval_gate --> production_monitoring["Production monitoring"]
  production_monitoring --> risk_and_compliance_assessment["Risk and compliance assessment"]
  risk_and_compliance_assessment --> cross_functional_ml_squad

Components

Data flow

A machine-learning artifact moves from delivery work into Ops approval, production deployment, monitoring, and iteration. As the organization matures, additional personas such as ML engineers and MLOps engineers maintain infrastructure, observability, security, and product delivery around the model lifecycle.

Why this architecture

The talk argues for product-oriented, cross-functional MLOps because production models continue evolving after training, critical infrastructure increasingly depends on ML, and LLMs amplify existing needs for monitoring, security, product thinking, and proportionate risk governance.

Models used: machine learning models · LLMs

Scale

• Automation, standardization, control, security, and observability should increase as MLOps requirements increase.
  “all of your automation standardization control security observability increases as your uh uh um mlops requirements also increase”transcript, stated
• As the number of machine-learning models increases, organizations may require machine-learning engineers or MLOps engineers.
  “as the number of machine learning models increase right you may actually require different personas involved right machine learning Engineers or even mlops Engineers”transcript, stated
• The speaker describes LLMs as causing massive accelerated adoption of product-team thinking, squads, and cross-functional virtual teams.
  “a massive massive like uh accelerated adoption of of uh again in this case more like like product team thinking so introduction of squads cross functional virtual teams”transcript, stated

Alternatives considered (roads not taken)

Treat ML as a finished project after initial delivery

Rejected because: The speaker argues that the model lifecycle begins once trained and in production, with future versions and system capabilities continuing to evolve.

Start with all MLOps complexity from day one

Rejected because: The talk recommends scaling automation, control, security, and observability with actual requirements rather than adopting maximum overhead immediately.

Deploy advanced AI everywhere regardless of risk or fit

Rejected because: The speaker says not everything must be solved with AI and some contexts are not viable because risk, explainability, or compliance overhead is too high.

Patterns

observability-tracing (operations)

The talk emphasizes production monitoring, SLOs, observability, and ML-specific metrics such as data-distribution monitoring.

human-in-the-loop-gate (governance)

Deployment risk can be mitigated through human-in-the-loop mechanisms and domain-expert alignment for higher-impact systems.

safety-policy-enforcement (risk and compliance)

The speaker frames security, compliance, ethical assessments, and proportionate risk as required extensions of production ML practice.

Tradeoffs

Move from project delivery to product operating model.

Chose Ongoing AI product lifecycle and cross-functional squads over One-time project completion.

Rationale: Production models continue changing after deployment and need product, engineering, domain, and operational ownership.

Scale MLOps overhead with maturity.

Chose Incremental automation, observability, security, and specialized personas over Maximum platform complexity from the start.

Rationale: The talk argues that team roles and platform controls should grow as model count and MLOps requirements grow.

Use proportionate risk to decide deployment readiness.

Chose Risk-based release with domain experts, SLOs, and optional human-in-the-loop mitigation over Uniform deployment rules for every AI use case.

Rationale: The same overhead is not viable for every project; high-impact systems need more controls, while sandboxed or mitigated systems may deploy with lighter guardrails.

Gotchas (what bit them)

Treating ML as a one-off project misses the real production lifecycle.

Root cause: The speaker says the model lifecycle starts after training and production deployment, when versions and capabilities continue changing.

Resolution: Adopt product thinking and cross-functional teams around ongoing ML system delivery.

Overbuilding MLOps from day one can add unjustified complexity.

Root cause: The talk ties automation, security, observability, and staffing needs to the number of models and maturity of production requirements.

Resolution: Increase automation, standardization, controls, security, and observability as MLOps requirements increase.

Deployment timing can fail by being too slow or too risky.

Root cause: MVP maturity decisions depend on the proportionate impact and risk of the use case.

Resolution: Use proportionate risk assessment, sandboxing, human-in-the-loop mitigation, and domain-expert alignment for higher-risk contexts.

Evaluation

The talk recommends production monitoring through SLOs, observability, traditional software metrics, ML-specific metrics, and proportionate risk assessment rather than reporting an evaluation harness for a single deployed system.

Metrics: SLOs for production performance. · Traditional software metrics plus ML-specific metrics such as data distribution monitoring. · Risk level and compliance demand as deployment-readiness inputs.

Outcomes

• The talk identifies LLMs as accelerating product-team thinking and cross-functional squads for production ML.
• It recommends mapping software best practices into ML rather than copying or discarding them wholesale.
• It frames proportionate risk as the decision rule for deploying AI systems responsibly.

This is an industry-state talk rather than a single deployment; it provides organizational and operating-model guidance, not product-specific throughput or adoption metrics.