Building an Enterprise LLMOps Stack: Lessons from Doordash

doordash · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLMOps platform design

DoorDash needed to reason about what an enterprise LLMOps stack should provide so teams could build GenAI applications with lower friction while managing model choice, inference, latency, cost, and proprietary/open-source tradeoffs.

Constraints:

High-QPS use cases make latency and cost first-order architecture concerns.
Enterprise teams need to support both proprietary and open-source LLMs.
The stack must reduce friction for application teams rather than just expose raw model APIs.
Cost controls, caching, inference, and serving choices have to be designed into the platform.

Architecture

The DoorDash talk frames LLMOps as an enabling stack with model options, inference and serving, caching, automation, tooling, and cost-aware controls. It is less a single deployed agent than a platform blueprint for helping teams build GenAI applications with minimum friction while supporting proprietary and open-source models.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  application-teams[Application Teams] --> llmops-stack[LLMOps Stack]
  llmops-stack --> model-layer[Model Layer]
  llmops-stack --> inference-serving[Inference Serving]
  inference-serving --> caching-layer[Caching Layer]
  llmops-stack --> automation-tooling[Automation Tooling]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Application Teams`	Teams building GenAI applications inside the company.	`LLMOps Stack`
`LLMOps Stack`	Shared platform layer for model access, inference, serving, caching, automation, and tooling.	`Model Layer`, `Inference Serving`, `Caching Layer`
`Model Layer`	Supports proprietary and open-source LLM options.	`LLMOps Stack`
`Inference Serving`	Runs model serving paths for latency- and QPS-sensitive workloads.	`LLMOps Stack`, `Caching Layer`
`Caching Layer`	Reduces latency and vendor-model cost for high-QPS use cases.	`Inference Serving`
`Automation Tooling`	Reduces developer friction across the GenAI application lifecycle.	`LLMOps Stack`

Data flow

Application teams send GenAI workloads through a shared LLMOps stack. The stack handles model choice, inference and serving, caching for high-QPS paths, and automation/tooling so teams can build with less friction and better cost and latency management.

“stack you need to figure out how to support both of these world because the needs are slightly different the infrastructure that's needed is going to be uh more challenging for one versus the others um what's really interesting is”transcript, stated

“with like minimum frictions so that means you got to focus a lot on automations toolings and at the same time as we know these large language model needs gpus and fine-tuning and inferencing be very very mindful about a costs managing”transcript, stated

Why this architecture

DoorDash emphasized platform-level LLMOps because the useful unit of value is not just calling a model; it is helping teams build faster while managing the operational dimensions of velocity, friction, and cost.

“velocity friction and cost how does this stack help with enable high velocity for team to build uh gen application Within theany with like minimum frictions so that means you got to focus a lot on automations toolings and at the same”transcript, stated

“scale QPS use cases those are costs and latency so that's very top of mind for those kind of use cas cases besides that um we know there's a you know proprietary LMS and open source LMS and as you think about building your”transcript, stated

Models used: proprietary LLMs · open-source LLMs · vendor LLMs

Scale

The talk explicitly targets high-QPS use cases where latency and cost dominate.
“scale QPS use cases those are costs and latency so that's very top of mind for those kind of use cas cases besides that um we know there's a you know proprietary LMS and open source LMS and as you think about building your”transcript, stated
The stack is intended for company-internal GenAI application teams, not a single one-off demo.
“stack help with enable high velocity for team to build uh gen application Within theany with like minimum frictions so that means you got to focus a lot on automations toolings and at the same time as we know these large language”transcript, stated

Alternatives considered (roads not taken)

Use only proprietary LLMs.

Rejected because: The stack needed to support both proprietary and open-source worlds.

“you know proprietary LMS and open source LMS and as you think about building your stack you need to figure out how to support both of these world because the needs are slightly different the infrastructure that's needed is going to”transcript, stated

Let teams integrate model APIs without shared tooling.

Rejected because: The platform goal was minimum friction through automation and tooling.

Call vendor LLMs directly for high-QPS workloads.

Rejected because: Caching can reduce both cost and latency for those paths.

“one that's making call to vendor L to help with like cost reduction as well as reduce latencies right um but you need you know a very interesting cache not your typical um KV cache so um need to figure that”transcript, stated

Patterns

`model-routing` (platform)

The stack is expected to support proprietary and open-source model choices.

“support both of these world because the needs are slightly different”transcript, stated

`cost-aware-routing` (operations)

High-QPS paths are designed with latency, cost, and caching in mind.

“help with like cost reduction as well as reduce latencies”transcript, stated

`semantic-caching` (performance)

Caching is raised as a way to reduce vendor calls for high-QPS LLM workloads.

“but you need you know a very interesting cache”transcript, stated

Tradeoffs

Model ecosystem

Chose Support proprietary and open-source LLMs over Standardizing on one LLM category.

Rationale: Different model ecosystems create different needs and should both be supported.

“support both of these world because the needs are slightly different”transcript, stated

High-QPS optimization

Chose Caching and cost-aware serving over Direct vendor calls for every request.

Rationale: Caching can reduce vendor-call cost and latency.

“making call to vendor L to help with like cost reduction as well as reduce latencies”transcript, stated

Gotchas (what bit them)

Cost and latency surface immediately at high QPS.

Root cause: Vendor model calls can be expensive and slow when request volume rises.

Resolution: Design caching and serving choices into the platform layer.

“scale QPS use cases those are costs and latency”transcript, stated

An LLMOps stack that ignores developer friction will not enable adoption.

Root cause: Teams need automation and tooling, not only abstract platform capabilities.

Resolution: Optimize the stack for velocity, friction reduction, and cost.

“velocity friction and cost how does this stack help with enable high velocity”transcript, stated

Evaluation

The source presents evaluation criteria rather than a deployed benchmark: the platform should be judged by velocity, friction, cost, latency, and support for model diversity.

Metrics: Velocity · Developer friction · Cost · Latency · QPS readiness

“it's all about velocity friction and cost”transcript, stated

Outcomes

The talk defines a platform-oriented LLMOps architecture rather than a single application outcome.
The reusable lesson is to treat cost, latency, model choice, and developer friction as first-class stack requirements.

The transcript discusses high-QPS use cases and internal teams building GenAI applications, but does not report deployed production volumes.

“highs scale QPS use cases those are costs and latency”transcript, stated