Scaling LLM Infrastructure: Building and Operating 24K GPU Clusters for LLaMA Training

meta · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: large-scale LLM training infrastructure

Meta needed to build and operate 24K-GPU clusters capable of training LLaMA 3 models for months despite hardware failures, thermal issues, network bottlenecks, and synchronized training dependencies.

Constraints:

• LLaMA 3 training used 15 trillion high-quality tokens and required thousands of GPUs for months.
• Synchronous training waits for the slowest GPU, so one slow or failed host can stall the whole job.
• The cluster needed dense power/cooling, high-throughput networking, topology-aware scheduling, and fast failure recovery.

Architecture

Meta adapted existing GPU chassis for higher-power training, built two 24K-GPU clusters using RoCE/Rocky and InfiniBand, organized the RoCE cluster as eight 3K-GPU domains, mapped tensor/pipeline/data parallelism to different network topology layers, tuned collective communication algorithms and load balancing, and measured training efficiency after checkpoint/failure overhead.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["LLaMA training job"] --> B["Parallelism planner"]
  A --> C["24K GPU cluster"]
  C --> D["3K GPU domains"]
  D --> E["GPU racks"]
  E --> F["Power and cooling system"]
  D --> G["Aggregation switching layer"]
  B --> H["Network topology-aware scheduler"]
  G --> H
  H --> I["Collective communication libraries"]
  F --> J["Hardware failure detection"]
  J --> K["Checkpoint and recovery system"]
  K --> C

Components

Data flow

Training data is split across data-parallel groups while model tensors and pipeline stages are partitioned across GPUs. Jobs run over 24K-GPU clusters composed of 3K-GPU domains, with tensor parallelism staying within servers, pipeline parallelism within small clusters, and data-parallel operations across small clusters. Checkpointing, failure detection, host removal, and recovery keep long-running synchronous training moving.

Why this architecture

The architecture follows the physical and communication shape of giant synchronous training: fit power/cooling, map parallelism to network topology, and reduce failure/recovery overhead because the whole job waits for slow devices.

Models used: LLaMA 3 · LLaMA 3 70B · PyTorch

Scale

• LLaMA 3 used 15 trillion high-quality tokens.
  “Lama 3 is at 15 trillion high quality tokens”transcript, stated
• The training jobs used thousands of GPUs for months.
  “incredibly large jobs requiring thousands of gpus that are actively used for months”transcript, stated
• Meta built two 24K-GPU clusters.
  “we decided to build both two 24K clusters one with Rocky another with infin ban”transcript, stated
• Each 24K cluster is stamped from eight 3K-GPU clusters.
  “in order to form a 24K GPU cluster we stamp eight of these 3K clusters within the data center building”transcript, stated
• The system reached about 95% effective training time on a 24-hour rolling window.
  “on a 24-hour rolling window basis we can use 95% of the time effectively”transcript, stated

Alternatives considered (roads not taken)

Use only one network fabric design

Rejected because: Meta built both RoCE/Rocky and InfiniBand clusters to learn from operational differences and inform future fabric direction.

Map parallelism without network topology awareness

Rejected because: Tensor, pipeline, and data parallel traffic have different bandwidth/latency needs and should align to server/small-cluster/cross-cluster layers.

Use the previous 500W ranking/recommendation chassis unchanged

Rejected because: Training required higher compute density, power, memory bandwidth, and thermal/mechanical changes.

Patterns

observability-tracing (production-operations)

Meta monitors training efficiency, thermal throttling, GPU failures, and network variability to identify root causes at cluster scale.

durable-execution (workflow)

Checkpointing and recovery are central to keeping long training jobs productive despite failures.

latency-optimization (infrastructure)

Network topology-aware placement maps communication patterns to the lowest-latency/highest-bandwidth layers available.

cost-aware-routing (infrastructure)

Power, cooling, density, and network choices are tuned to extract usable compute from 24K-GPU clusters.

Tradeoffs

Network fabric strategy

Chose build both RoCE/Rocky and InfiniBand clusters over standardizing immediately on one fabric.

Rationale: Meta wanted operational experience from both to inform future generative-AI fabrics.

Chassis design

Chose adapt a ranking/recommendation chassis for higher-power GPUs over waiting for a fully new platform.

Rationale: The fast-moving GenAI race required urgent adaptation, but it created power/cooling/mechanical work.

Topology mapping

Chose map parallelism types to network layers over treating the 24K cluster as uniform bandwidth.

Rationale: The 24K cluster has different bandwidth/latency regions, so communication patterns need placement awareness.

Gotchas (what bit them)

One thermally throttled GPU cut training performance by 50%.

Root cause: Synchronous all-reduce made the whole job wait for the slowest GPU.

Resolution: Remove the host and tune automation thresholds.

GPU failure incidence rises as job size grows.

Root cause: More GPUs reduce mean time between failures and can halt the whole synchronized job.

Resolution: Invest in hardware reliability, vendor mitigations, checkpointing, and fast recovery.

High-density GPU clusters create power and cooling constraints beyond the GPU itself.

Root cause: Moving from 500W to 700W GPUs required rack, row, fan, dB, and thermal design changes.

Resolution: Redesign chassis internals, supporting services placement, and data-center density layout.

Large clusters performed poorly out of the box.

Root cause: Network bandwidth, latency, collective algorithms, and load balancing were not automatically aligned to training communication patterns.

Resolution: Optimize parallelism placement, collectives, load balancing, and topology-aware software layers.

Evaluation

Meta evaluates the cluster by training efficiency over rolling windows, performance degradation, failure incidence, checkpoint/recovery overhead, network benchmarks, roofline proximity, and application performance.

Metrics: training efficiency · 24-hour rolling effective time · performance degradation · failure incidence · time lost from last checkpoint · network bandwidth · latency · roofline performance

Outcomes

• Meta built two 24K-GPU clusters for LLaMA training.
• The cluster reached about 95% effective training time on a 24-hour rolling window.
• Network topology-aware mapping and custom collectives improved large-cluster performance near roofline.
• Thermal throttling and GPU failure handling became explicit operational disciplines.

This is a training infrastructure case rather than inference or agent application, but its durable-execution, observability, and topology lessons recur in large agent platforms.