Scaling LLM Inference Infrastructure at Meta: From Model Runner to Production Platform

meta · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: hyperscale LLM inference

Meta needed to serve LLMs for public and internal products at massive scale while controlling latency, throughput, reliability, cost, and developer experience.

Constraints:

LLM inference combines prefill and decode phases with different compute, memory bandwidth, and KV-cache demands.
Large models require tensor and pipeline parallelism to fit across GPUs and hosts.
Production workloads lose theoretical FLOPS through latency bounds, workload variability, buffers, and operational overhead.

Architecture

Meta's inference platform combines specialized LLM runtimes with continuous batching and KV cache, tensor/pipeline/context parallelism for fitting models, disaggregated prefill/decode services, hierarchical KV-cache management across HBM/DRAM/flash, cache-aware product/data-flow design, quantization and domain-specific optimizations, and observability/automation across many deployment tails.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Product business logic"] --> B["Safety and retrieval services"]
  B --> C["LLM runtime"]
  C --> D["Prefill service"]
  D --> E["Decode service"]
  E --> F["Hierarchical KV cache"]
  F --> G["Sticky routing"]
  D --> H["Tensor and pipeline parallelism"]
  C --> I["Observability and automation"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Product business logic`	Application layer including safety checks, retrieval, function calls, and orchestration around the LLM.	`LLM runtime`, `Safety and retrieval services`
`Safety and retrieval services`	Pre/post model checks, search, and business logic that add end-to-end latency.	`Product business logic`, `LLM runtime`
`LLM runtime`	Serving runtime with continuous batching and KV cache support.	`Product business logic`, `Prefill service`, `Decode service`, `Hierarchical KV cache`
`Prefill service`	Compute-heavy first-token phase that processes prompt context.	`LLM runtime`, `Decode service`, `Tensor and pipeline parallelism`
`Decode service`	Memory-bandwidth-heavy token generation phase.	`Prefill service`, `Hierarchical KV cache`
`Hierarchical KV cache`	HBM/DRAM/flash storage hierarchy for reused system prompts and chat history.	`LLM runtime`, `Decode service`, `Sticky routing`
`Sticky routing`	Consistent hashing that routes same-session retries to hosts with the relevant cache.	`Hierarchical KV cache`, `Product business logic`
`Tensor and pipeline parallelism`	Model partitioning methods to fit 70B and 405B class models across GPUs and hosts.	`Prefill service`
`Observability and automation`	Measures tail deployments and finds low-hanging GPU/cost optimization opportunities.	`LLM runtime`

Data flow

Requests enter product business logic, safety and retrieval steps, then LLM inference. Prefill processes the prompt and decode generates tokens while KV cache is retained. Large models may be partitioned across GPUs/nodes, prefill and decode may run in separate services with KV transfer between them, session requests are sticky-routed to hosts with cache, and product teams co-design chat-history management to improve cacheability.

3 supporting quotes

“We call the generation for the first token, prefill, and all the tokens after the first, decode.”article, stated

“We replicate the already partitioned weights, and we run prefill and decode in different services.”article, stated

“we co-designed a chat history management with our product team, and we even further customized the request data flow”article, stated

Why this architecture

Meta treats each model as a system: product requirements, model shape, hardware resources, caching, routing, and deployment management all have to be jointly optimized.

3 supporting quotes

“LLM serving is probably the most interesting model serving problem that I work on because a model is a system.”article, stated

“The best solution really requires you to think comprehensively to get some joint optimization between model, product, and system.”article, stated

“making LLM faster is really about fitting this LLM math operation within these three types of system resources”article, stated

Models used: Meta AI models · Llama family models · 8B models · 70B models · 405B models · ancillary safety/planning/rewarding/function-calling models

Scale

During busy RLHF weeks, Meta processed hundreds of millions of examples.
“During the busy week of RLHF, our team has to process hundreds of millions of examples.”article, stated
A 405B bf16 model takes over 800 GB of weights and needs two nodes.
“The 405B model can't fit in 8 H100. The weights is going to take over 800 gig under bf16, so you will need two nodes.”article, stated
Disaggregation can remove 10x P99 decode latency caused by long prefill.
“get rid of this annoying 10x P99 latency for decode, because that P99 is basically the average of your prefill latency.”article, stated
KV caching commonly reduces latency and capacity by more than 50%.
“It's very common for us to see over 50% of reduction for both latency and capacity.”article, stated
Meta reports a 10x foundation from basics and another 2x-4x from distributed inference, smaller inputs/models, and caching.
“Getting the basics right will give you a 10x foundation. Distributed inference, smaller input and model, and caching will give you another 2x to 4x boost.”article, stated

Alternatives considered (roads not taken)

Use training or eval code directly for inference serving

Rejected because: Production serving needs LLM-specific runtimes, batching, KV cache, and hardware-aware parallelism.

“don't just grab your training or eval code. Find some runtime specialized in LLM inference serving.”article, stated

Rely only on adding more GPUs

Rejected because: Throwing capacity blindly does not address bottlenecks in prefill/decode, caching, and product-data flow.

“Throwing capacity blindly only takes you this far, not that far.”article, stated

Disaggregate prefill/decode without accounting for KV transfer and deployment complexity

Rejected because: Disaggregation adds hundreds of MB of KV transfer and requires scheduling, overlap, rollout, and weight-update work.

“Between prefill and decode, you have to transfer hundreds of megabytes of KV cache.”article, stated

Patterns

`latency-optimization` (production-operations)

Meta decomposes inference latency into prefill, decode, KV transfer, business logic, and routing bottlenecks, then optimizes each layer.

“When you look at the end-to-end latency, inference a lot of time is just a small portion of it. If you want to make good use of KV cache, and the persistent KV cache, you often have to do product and infra co-optimization.”article, stated

`semantic-caching` (inference)

Hierarchical KV caching stores reusable system prompts and chat history in HBM, DRAM, or flash depending on activity.

“We can build a hierarchical caching system that actually matches the KV cache usage pattern. For example, the common system prompt might get cached in the HBM.”article, stated

`model-routing` (model-operations)

Different model sizes, ancillary models, and hardware placements are selected based on product latency, quality, and cost needs.

“A mature product typically involves serving a handful of LLMs. You get the main big model for your chat, and you get a bunch of ancillary models for safety, planning, rewarding, function calling, to complete the end-to-end loop.”article, stated

`observability-tracing` (production-operations)

Observability and automation identify tail deployments that collectively consume more GPUs than the main model.

“observability and automation are no longer nice to have, they directly help you to claim all of these low-hanging fruits.”article, stated

`context-engineering` (context-management)

Meta co-designs chat history and request data flow to improve cache hit rate and leave more latency budget.

“To improve the cache hit rate, you not only need a big cache, as we discussed how you can build a hierarchy, you also need prompts that are very cacheable.”article, stated

Tradeoffs

Prefill/decode deployment

Chose disaggregated services over one continuous-batching service for all phases.

Rationale: Disaggregation separates compute-heavy prefill from memory-bandwidth-heavy decode but adds KV-transfer and deployment complexity.

“by doing disagg, you are basically signing up for these hard optimizations, like request scheduling, and overlap data transfer and compute”article, stated

Cache hit-rate strategy

Chose product and infra co-optimization over simply adding a bigger KV cache.

Rationale: Cacheability depends on prompt/request shape and session routing, not only cache size.

“If you want to make good use of KV cache, and the persistent KV cache, you often have to do product and infra co-optimization.”article, stated

Reliability/cost target

Chose choose only two or three among quality, latency, throughput, reliability, and cost over maximizing every dimension simultaneously.

Rationale: Stricter reliability or speed targets can multiply cost and must match product needs.

“Out of quality, latency, throughput, reliability, and cost, you can probably only get to choose two or three of them.”article, stated

Gotchas (what bit them)

Long prefill can stall decode latency.

Root cause: Continuous batching mixes prefill into decode steps, so a 32K input can block existing generation.

Resolution: Disaggregate prefill and decode into separate services.

“Imagine you suddenly get a request with 32K input, all of the decoding will get stuck for a few seconds, and your customer will actually see it.”article, stated

KV cache is hard to hit with naive chat history prompts.

Root cause: Positional encoding means the same tokens in different positions produce different embeddings.

Resolution: Co-design chat history management and sticky routing with product teams.

“The same tokens showing up in the beginning of the sentence versus the end of the sentence will have different embeddings.”article, stated

Production deployments can lose far more FLOPS than benchmarks suggest.

Root cause: Latency bounds, operating buffer, short inputs, and workload variability reduce effective throughput.

Resolution: Track product-relevant signals and optimize deployment tails, not just ideal kernel benchmarks.

“It's very common to lose 50% effective FLOPS, even at the earliest kernel benchmarking stage, especially for shorter input.”article, stated

Evaluation

Meta evaluates inference by latency SLOs, achievable throughput, effective FLOPS, GPU and streaming-processor utilization, cache-hit impact, product signals, and cost-saving/dev-experience outcomes.

Metrics: latency SLO · P99 latency · TTFT · TTIT · effective FLOPS · GPU utilization · tokens per second · latency/capacity reduction

“The exact GPU device utilization and the streaming processors utilization really depends on your workload. Typically, we do see them go up, like go up to more than 50% very easily for our workload.”article, stated

“don't forget to continuously evaluate and test all of these acceleration techniques directly using the signals from your product.”article, stated

Outcomes

Meta inference supports Meta AI, smart glasses, RLHF, data curation, and distillation workloads.
Basic runtime choices provide a 10x foundation.
Distributed inference, smaller inputs/models, and caching add another 2x-4x boost.
KV caching commonly reduces latency and capacity by more than 50%.

The talk is candid that theoretical capacity is misleading in production; real gains require workload-specific profiling and product/infra co-design.

“Our team optimize inference backend for Meta AI and smart glasses. Apart from the public-facing traffic that everybody sees, we also have a lot of internal traffic, like RLHF, data curation, distillation that goes into the making of Llama family of models.”article, stated

“Getting the basics right will give you a 10x foundation. Distributed inference, smaller input and model, and caching will give you another 2x to 4x boost.”article, stated