Scaling AI Infrastructure: From Training to Inference at Meta

meta · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: hyperscale AI training and inference infrastructure

Meta needed to scale AI infrastructure from large training jobs to distributed inference while GPU counts, model sizes, power demands, failure rates, and time-to-market pressure were all increasing rapidly.

Constraints:

• Training clusters grew from hundreds of GPUs to over a hundred thousand GPUs in roughly two years.
• Large models no longer fit into the GPU memory of a single host, creating distributed-inference placement and routing constraints.
• SPMD pre-training has an all-or-nothing failure model where one host failure can stop the job and force checkpoint restore.
• Data centers and utilities must handle gigawatt-scale facilities and 10-15 MW power oscillations from training jobs.
• AI products need hallucination reduction, secure personalized context, and fast infrastructure delivery.

Architecture

Meta describes a full-stack AI infrastructure strategy spanning secure code-execution sandboxes, distributed inference, RAG/vector databases, graph RAG, a possible shift from SPMD to single-controller programming, faster restarts and checkpointing, huge GPU clusters, new data-center designs, nuclear-power procurement, and power-smoothing work. The architecture is guided by optionality, time to market, and innovation rather than a single fixed cluster pattern.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["GPU training clusters"] --> B["Checkpoint and restart layer"]
  B --> C["Single-controller programming model"]
  A --> D["Power management systems"]
  D --> E["Data centers"]
  F["Distributed inference scheduler"] --> G["Model partitions"]
  G --> H["GPU backend network"]
  I["RAG and vector database layer"] --> J["LLM serving layer"]
  K["Graph RAG layer"] --> I
  L["Secure code execution platform"] --> M["AI software stack"]

Components

Data flow

AI workloads move through training, post-training, secure code execution, distributed inference, and retrieval-augmented response generation. Large models are partitioned across hosts, scheduled with GPU-to-GPU backend-network placement constraints, and served to products. Retrieval paths vectorize queries, retrieve personalized context from vector or graph stores, and feed it to LLMs. Training jobs use checkpoint/restart mechanisms and new control models to reduce the impact of host failures and power oscillations.

Why this architecture

Meta's architecture keeps optionality at every layer because product requirements, model sizes, hardware, and power constraints are changing faster than traditional infrastructure cycles. Distributed inference addresses model-memory limits, RAG and graph RAG address hallucinations, checkpoint/single-controller work addresses failure and power oscillation costs, and new data-center/power strategies address compute growth.

Models used: Llama 3 · Llama 4 · LLMs · vector databases · graph RAG

Scale

• Meta says a large training job was about 256 GPUs two to three years earlier.
  “about two to three years ago, at least for us, the state-of-the-art for a large training job was about 256 GPUs.”transcript, stated
• Meta built two 24,000-GPU clusters to train Llama 3.
  “with LAMA three, we announced that we built two 24,000 GPU clusters to train LAMA three”transcript, stated
• Meta announced a single Llama 4 cluster with over a hundred thousand GPUs.
  “with LAMA four, just last year around October, we announced that we had a single cluster with over a hundred thousand GPUs in it.”transcript, stated
• Meta characterizes the jump from 256 GPUs to a hundred thousand GPUs as a 38,000% increase.
  “think about that 256 GPUs to a hundred thousand GPUs. That's a 38000% increase in just over two years.”transcript, stated
• Meta expected over a million GPUs online internally by the end of the year.
  “we've already announced that internally that will have over a million GPUs online by the end of the year.”transcript, stated
• Meta describes a new two-gigawatt facility roughly the size of Manhattan.
  “This is a new two gigawatt facility that's physically about the size of Manhattan.”transcript, stated
• Meta issued an RFP for one to four gigawatts of new nuclear power generation.
  “we came up with an RFP seeking one to four gigawatts of new nuclear power generation in the United States.”transcript, stated
• Meta reports the secure code execution platform has run more than 40 billion instances since launch.
  “we have more than 40 billion plus code execution instances that have been executed since it was launched in January.”transcript, stated
• Meta says a training job can create 10-15 MW power oscillations.
  “It can be on the order of 10 if not 15 megawatts of power.”transcript, stated

Alternatives considered (roads not taken)

Single-host inference

Rejected because: Large models no longer fit in one host's GPU memory.

Pure SPMD programming model

Rejected because: A single host failure halts the entire pre-training job and forces costly checkpoint restore.

Rely only on model memory for answers

Rejected because: Hallucinations remain a major problem and retrieval can provide additional context.

Continue one-size-fits-all 150 MW regions

Rejected because: AI scale pushed Meta toward new options such as two-gigawatt facilities and nuclear-power procurement.

Patterns

rag-retrieval (retrieval)

Meta uses vector databases and graph RAG to retrieve external and personalized context for LLM responses.

privacy-preserving-processing (governance)

Meta frames personalized context retrieval as secure and privacy safe.

durable-execution (reliability)

Meta uses faster restarts, checkpointing, and single-controller exploration to reduce all-or-nothing training failures.

latency-optimization (infrastructure)

Distributed inference adds routing and placement constraints so large model partitions can serve queries across hosts.

batch-offline-processing (infrastructure)

Meta's secure code execution system runs huge numbers of isolated code-execution instances for AI workflows.

Tradeoffs

Programming model

Chose single-controller exploration over remaining solely with SPMD.

Rationale: SPMD's all-or-nothing failures become more costly as clusters grow.

Hallucination mitigation

Chose RAG, vector search, and graph RAG over only relying on the LLM's own memory.

Rationale: External context can make answers more accurate while preserving secure personalized retrieval.

Data-center strategy

Chose new facility shapes and energy procurement over only stamping out existing 150 MW regions.

Rationale: GPU growth changed power and time-to-market requirements beyond the old region template.

Gotchas (what bit them)

Inference now has training-like placement constraints.

Root cause: Large models must be partitioned across hosts and those tasks need GPU backend-network connectivity.

Resolution: Meta treats distributed inference placement and routing as a scheduling-layer problem.

SPMD failure semantics waste GPU cycles at large scale.

Root cause: A single host failure can halt pre-training and force expensive checkpoint restore.

Resolution: Meta is exploring single-controller programming plus faster restarts and checkpointing.

Training jobs can stress the electrical grid.

Root cause: Job starts and stops create instantaneous 10-15 MW power changes rather than linear ramps.

Resolution: Meta combines faster restarts, faster checkpointing, and single-controller work to smooth peaks and valleys.

Physical infrastructure planning cannot keep a single old template.

Root cause: AI compute growth pushes facilities from 150 MW regions toward gigawatt-scale sites and new energy procurement.

Resolution: Meta explores two-gigawatt regions and one-to-four-gigawatt nuclear-power RFPs.

Evaluation

Meta evaluates the infrastructure through scale, placement constraints, reliability/failure behavior, time to market, power behavior, and product-serving needs rather than a single model-quality benchmark.

Metrics: 256 GPUs to over 100,000 GPUs · 38,000% increase in just over two years · Over a million GPUs planned internally by year end · Two 24,000-GPU clusters · Two-gigawatt facility · One-to-four-gigawatt nuclear RFP · 250,000 code-execution instances together · 40 billion-plus code-execution instances · 10-15 MW power oscillations

Outcomes

• Meta scaled from 256-GPU large jobs to clusters with over a hundred thousand GPUs.
• Meta built two 24,000-GPU clusters for Llama 3 and announced over a million internal GPUs by year end.
• The secure code execution platform reached 250,000 concurrent instances and more than 40 billion executed instances.
• Meta is using vector DBs and graph RAG to reduce hallucinations with external context.
• Meta is exploring single-controller programming, faster restarts, and faster checkpointing to mitigate failures and power swings.
• Meta is planning gigawatt-scale data centers and nuclear-power procurement for AI growth.

The talk reports a shift from 256-GPU jobs to two 24,000-GPU clusters, then over 100,000 GPUs, a 38,000% increase, planned one-million-GPU internal capacity, two-gigawatt facilities, a one-to-four-gigawatt nuclear RFP, 250,000 secure code-execution instances together, 40 billion-plus instances, and 10-15 MW training-job power oscillations.