Evolving ML Infrastructure for Production Systems: From Traditional ML to LLMs

doordash · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production ML and LLM infrastructure

DoorDash and similar ML organizations need to integrate LLMs into production systems without treating them as universal replacements for traditional ML, because LLMs introduce higher error rates, new call patterns, cost, latency, explainability, and platform-flexibility constraints.

Constraints:

LLMs are wrong a significant amount of the time, including wrong format and ambiguous responses.
User-visible errors and high-volume items raise the risk of relying exclusively on LLMs.
Many ML platforms were optimized for trained-in-platform models, feature stores, ranking, and recommendation call patterns.
LLM applications need context windows, prompts, RAG, evaluation stores, and other shared resources rather than classic feature-only serving.
Traditional models may still win on cost, latency, interpretability, and specific performance metrics.
Platform teams need enough flexibility to track fast-changing open-source and foundation-model ecosystems.

Architecture

The talk describes a hybrid ML infrastructure strategy rather than a single DoorDash product: use LLMs for low-risk or bootstrapping cases, wrap them with guardrails and traditional ML validators, treat LLM output as model features or fallbacks, preserve traditional models where latency/cost/explainability matter, and evolve platform layers to support bring-your-own models, context/prompt/data stores, open-source tooling, and separation between user code and infrastructure wrappers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  use-case-risk-assessment["Use-case risk assessment"] --> llm-task-layer["LLM task layer"]
  use-case-risk-assessment --> traditional-ml-guardrail-layer["Traditional ML guardrail layer"]
  llm-task-layer --> traditional-ml-guardrail-layer
  llm-task-layer --> prompt-and-context-resources["Prompt and context resources"]
  traditional-ml-guardrail-layer --> custom-model-path["Custom model path"]
  custom-model-path --> evaluation-baseline["Evaluation baseline"]
  prompt-and-context-resources --> platform-abstraction-layer["Platform abstraction layer"]
  open-source-model-integration["Open-source model integration"] --> platform-abstraction-layer
  platform-abstraction-layer --> evaluation-baseline
  cross-functional-platform-partnership["Cross-functional platform partnership"] --> use-case-risk-assessment
  cross-functional-platform-partnership --> platform-abstraction-layer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Use-case risk assessment`	Determines whether LLM error cost is acceptable for a given product or infrastructure workflow.	`LLM task layer`, `Traditional ML guardrail layer`
`LLM task layer`	Handles low-risk, fallback, bootstrapping, or creative generation workloads such as long-tail labeling.	`Traditional ML guardrail layer`, `Prompt and context resources`
`Traditional ML guardrail layer`	Validates LLM outputs or consumes them as features for more robust models.	`LLM task layer`, `Custom model path`
`Custom model path`	Uses tree-based, DNN, or simpler models when latency, explainability, or metric improvements justify traditional ML.	`Traditional ML guardrail layer`, `Evaluation baseline`
`Prompt and context resources`	Provide prompts, context windows, RAG-style context, datasets, and evaluation stores needed for LLM workflows.	`LLM task layer`, `Platform abstraction layer`
`Open-source model integration`	Allows teams to pull in current foundation models and tools instead of being locked to internal-only models.	`Platform abstraction layer`, `LLM task layer`
`Platform abstraction layer`	Separates user code from infrastructure wrappers so the architecture can evolve quickly.	`Prompt and context resources`, `Open-source model integration`, `Evaluation baseline`
`Evaluation baseline`	Lets teams quickly judge whether ML or LLM use is viable before investing in robust production models.	`Custom model path`, `Platform abstraction layer`
`Cross-functional platform partnership`	Aligns data scientists, ML engineers, infrastructure teams, and product teams on constraints and tradeoffs.	`Use-case risk assessment`, `Platform abstraction layer`

Data flow

A product team starts by assessing error cost and use-case risk, then routes low-risk or bootstrapping tasks to an LLM while using traditional models as guardrails, validators, or downstream consumers of LLM output. Platform infrastructure must load prompts and context, support external/open-source models, provide evaluation and dataset stores, and let teams replace pieces as the model ecosystem changes.

“if you're trying to integrate llms uh into a product or or Surface uh you need to understand the the limitations uh for where we currently are with l”transcript, stated

“you could also you know view maybe some more traditional uh models acting as guard rails to the outputs of of of some llms”transcript, stated

Why this architecture

The hybrid strategy follows from the speaker's view that LLMs are useful but error-prone, expensive, and operationally different from classic ML. Keeping traditional models in the loop preserves latency, explainability, and robustness where needed, while flexible platform abstractions let teams adopt open-source tools and new model patterns without rewriting the stack.

“just because you can use an llm for something doesn't mean you should right LMS are of magnitude more expensive”transcript, stated

“separating user code uh from like uh the code related to infrastructure and the related wrappers um allows you to really evolve your architecture very quickly”transcript, stated

Models used: LLMs · Traditional ML models · Tree-based models · DNNs · Feature stores · RAG-based architectures · Context windows · Prompt management stores · Dataset stores · Evaluation stores · Metaflow

Scale

Large-company ML platforms had previously depended on massive compute and data advantages.
“five years right even not even that long ago uh if you wanted to do anything in ml you had to be at a large company”transcript, stated
Classic ML platforms often optimize for millions of QPS.
“most of them are built around that that hyper optimization like how can we serve you know millions of of QPS”transcript, stated
At Uber, the speaker saw a task that took less than half a day at Uber take two weeks at Meta.
“less than half a day at Uber would take two weeks uh at meta right”transcript, stated
Platform teams need to be six months to a year ahead of product teams.
“the platform team needs to be you know six months to a year ahead of uh you know the business essentially the the product teams”transcript, stated

Alternatives considered (roads not taken)

Use LLMs for every ML problem because they are available.

Rejected because: The speaker argues that error rate, cost, and use-case fit must decide whether an LLM belongs in the system.

“llms aren't like an oracle right they are wrong uh significant amount of time”transcript, stated

Rely exclusively on LLMs for user-visible labels or high-volume items.

Rejected because: Even low-risk labeling needs guardrails when labels are user-visible or apply to popular items.

“if it's like on a very popular item right uh you don't want to exclusively rely on llms”transcript, stated

Keep ML platforms limited to models trained on the internal platform.

Rejected because: A platform that cannot bring in external or open-source models is disadvantaged as the ecosystem changes quickly.

“they only serve models that have been trained on their platform right and run off you're at a massive disadvantage”transcript, stated

Use LLMs where traditional models can satisfy the use case.

Rejected because: Traditional models may be cheaper, more interpretable, faster, or better for specific metrics.

“if it makes sense for you to like uh like business sense where even like a small changes in um you know Au or or performance metrics uh matter”transcript, stated

Patterns

`guardrails-io-filtering` (hybrid ML validation)

Traditional models can act as guardrails over LLM outputs before those outputs reach users or downstream workflows.

“more traditional uh models acting as guard rails to the outputs of of of some llms”transcript, stated

`fallback-degradation` (low-risk LLM use)

The speaker suggests using an LLM as a fallback when no stronger model exists, then feeding that result into more robust models when needed.

“it could be a great fallback right if you don't have anything else call an LM see what it thinks right or use it as an input”transcript, stated

`cost-aware-routing` (model choice)

Teams should choose LLMs only when their capability justifies the added API, hosting, workflow, and latency costs.

“the complexity they actually introduce is incredibly costly not just in terms of the cost of pinging apis or self-hosting models but in terms of the workflows you need to develop”transcript, stated

`eval-driven-development` (baseline validation)

Teams are advised to bootstrap quickly and evaluate whether ML or LLM use is viable before investing in a robust production model.

“if you're just trying to establish a baseline um you want like a a quick and dirty way to uh to really evaluate the the use of ml in your system”transcript, stated

Tradeoffs

Where LLMs should enter production ML systems.

Chose Low-risk, fallback, or feature-input roles over Exclusive reliance on LLM outputs.

Rationale: LLMs can bootstrap useful functionality, but user-visible and high-impact errors still need validation or traditional-model support.

“maybe you're using llms to quickly add uh tags to to restaurants uh or something like that right where it's very user visible the cost of maybe mislabeling one item”transcript, stated

Whether to keep investing in traditional ML models.

Chose Use traditional models alongside LLMs over Replacing tree-based, DNN, or simple models wholesale.

Rationale: Traditional models often remain good enough and can outperform LLMs on latency, explainability, or specific metrics.

“you'd find that a majority of the models are actually tree based b or or very simple models right um uh and for a lot of use cases that that's like good enough”transcript, stated

How flexible the platform layer should be.

Chose Open-source-friendly, evolvable abstractions over Highly constrained internal-only serving stacks.

Rationale: The model and infrastructure ecosystem changes too quickly for rigid platforms that cannot swap models and tools.

“build in a way that uh you can evolve um uh because things will change”transcript, stated

Gotchas (what bit them)

LLM integration fails when teams underestimate model error modes.

Root cause: LLMs can be wrong in content, format, and ambiguous behavior, not just hallucinated facts.

Resolution: Assess error cost before choosing use cases and keep guardrails around user-visible outputs.

“they have a very high uh error rate not just in terms of like maybe giving the the wrong responses but maybe even the format is is not even what what you might expect”transcript, stated

Existing ML serving stacks may not fit LLM application patterns.

Root cause: Traditional ML stacks assume feature-store and transactional ranking/recommendation call patterns.

Resolution: Add prompt, context, dataset, evaluation, and RAG-oriented resources at the platform level.

“there's really no concept of a feature store anymore right there's like you know rag based architectures and context windows”transcript, stated

Internal-only model platforms can fall behind the open-source ecosystem.

Root cause: Open-source models and tools are changing faster than some large-company internal platforms can absorb.

Resolution: Design infrastructure to pull in the latest foundation models and open-source tools with low effort.

“if your stack is not able to easily pull in you know the latest foundational models or if that requires a lot of effort uh that's going to be like a nightmare”transcript, stated

Platform teams build poor ML tooling if they never use their own product.

Root cause: Infrastructure engineers who never train or deploy models lack empathy for data scientists and ML engineers.

Resolution: Dogfood the tooling and involve platform partners early in product-team modeling decisions.

“if you're building an and you never trained a model before right or never deployed the model before uh it's going to be very hard for you to one empathize”transcript, stated

Evaluation

The talk recommends evaluating LLM adoption by error cost, product visibility, baseline viability, latency, cost, explainability, and platform constraints rather than using a single model-quality metric.

Metrics: Error cost and user visibility · Latency · API or self-hosting cost · Explainability/interpretablity need · Baseline viability · Platform integration effort

“if the the cost of an error is negligible um so you could imagine you know maybe you're using llms to quickly add uh tags”transcript, stated

“they care more about latency than um maybe like even correctness”transcript, stated

Outcomes

The case does not report a single deployed DoorDash LLM system; it documents production-infrastructure design lessons from DoorDash and other large ML organizations.
The recommended operating model is hybrid: LLMs, traditional models, guardrails, flexible platform infrastructure, and cross-functional platform partnership.
The speaker's final advice is to design for change as model and infrastructure ecosystems evolve.

This is a production-ML infrastructure discussion rather than a product launch report; quantitative claims include millions-of-QPS platform patterns, a six-month-to-one-year platform planning horizon, and large-company experience across Netflix, Meta, Uber, DoorDash, and Yahoo.

“if I could you know summarize in one sentence uh designed for change”transcript, stated

“worked at Netflix as I said meta Uber door Dash um and Yahoo”transcript, stated