Scaling AI Network Infrastructure for Large Language Model Training at 100K+ GPU Scale

meta · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: 100K+ GPU AI training network infrastructure

Meta needed to evolve its AI network from a 24K GPU single-building cluster to a 100K+ GPU multi-building region while preserving training throughput and operational reliability under congestion, long-distance, PFC, buffer, and firmware issues.

Constraints:

• Llama 4 required more GPU scale than the 24K GPU Llama 3 cluster.
• The 100K+ cluster spans five buildings, 20 data halls, and more than 3 km.
• AI traffic is bursty and creates congestion at different layers of the network.
• Lossless RoCE traffic depends on PFC, which can create head-of-line blocking and buffer contention.
• Some failures only reproduce at large GPU scale, such as an 8K GPU job across multiple data centers.

Architecture

Meta's network evolved from a 24K GPU, one-building Llama 3 cluster to a 100K+ GPU, five-building regional cluster. The 24K design used a CRAM approach with two AI zones per data hall and an ATSW super-spine layer. The 100K+ design scales that pattern across 38 AI zones, supporting services, CTSW/RTSW layers, deep-buffer switches, aggressive PFC threshold tuning, low-latency networking, dashboards for all-layer health metrics, queue fixes for RoCE control packets, transient-congestion mitigation, and firmware debugging.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["100K+ regional cluster"] --> B["AI zones"]
  B --> C["RTSW switches"]
  C --> D["CTSW switches"]
  B --> E["Supporting services"]
  D --> F["Lossless RoCE traffic"]
  G["Monitoring dashboards"] --> A
  G --> C
  G --> D
  H["Firmware and optics stack"] --> A
  I["24K Llama 3 cluster"] --> J["ATSW super-spine"]
  I --> B

Components

Data flow

Training data flows from storage/supporting services into AI zones and GPUs across a regional training network. Within each building, GPU servers connect through RTSWs and CTSWs; across buildings, aggregation layers connect many data halls. Monitoring dashboards expose health metrics, while network fixes move control traffic into lossless queues, tune buffers/PFC, and mitigate transient congestion so training jobs can continue ramping without downtime.

Why this architecture

Meta packed GPUs into AI zones to maximize building capacity for Llama 3, then moved to a multi-building regional design when next-generation models demanded more scale. The network architecture adds deep buffers, PFC tuning, health dashboards, and fleet-safe rollout practices because congestion and firmware issues can otherwise degrade large training jobs in ways that only appear at 8K-to-100K GPU scale.

Models used: Llama 3 · Llama 4 · generative AI training workloads

Scale

• Meta built the 24K cluster in 2023 for Llama 3 training.
  “This is the what we call the 24K cluster. We built this network in 2023 specifically for training llama 3.”transcript, stated
• The 24K cluster used one building with four data halls.
  “the network occupies one data center building of four data holes.”transcript, stated
• The 24K cluster resulted in 24,000 interconnected GPUs.
  “This aggregation tier resulted in 24,000 interconnected GPUs.”transcript, stated
• The 100K+ cluster spans five buildings and 20 data halls.
  “This is a type one region comprising five buildings. Each building houses four data holes, bringing the total to 20 data holes across the entire region.”transcript, stated
• The 100K+ cluster has 38 AI zones and over 100,000 GPUs.
  “In total here we have 38 AI zones bringing the total scale of GPUs to over 100,000.”transcript, stated
• The 100K+ cluster spans more than 3 km.
  “there are five DCs in in the 100k plus cluster which are spanning over 3 km.”transcript, stated
• An 8K GPU job across multiple DCs exposed a 30% degradation in one DC.
  “we were running an 8K GPU job which was running across multiple DCs but then one of the DCs which was DC number one actually showed 30% degradation of performance.”transcript, stated
• Meta's next Prometheus milestone may cross a million GPUs and train over distances beyond 100 km.
  “We expect that the GPU scale will become in the hundreds of thousands, may even cross a million.”transcript, stated

Alternatives considered (roads not taken)

Keep the 24K single-building design

Rejected because: Llama 4 and next-generation models demanded greater GPU scale.

Let PFC propagate through congestion

Rejected because: Large queue buildups and head-of-line blocking would harm model performance.

Treat lossy RoCE control drops as harmless

Rejected because: Control-packet loss degraded workload performance even though lossy traffic was under 1% of traffic.

Roll congestion fixes fleet-wide without staged monitoring

Rejected because: Meta deployed the fix in one DC first, monitored side effects, and then rolled it out without downtime.

Patterns

latency-optimization (infrastructure)

Meta tuned buffers, PFC thresholds, low-latency networking, and control-traffic queues to protect training throughput.

observability-tracing (operations)

Meta built dashboards with network-health metrics at all layers to debug high-scale training problems.

canary-rollout (deployment)

Meta deployed the congestion fix in one data center, monitored side effects, and then rolled it out fleet-wide without downtime.

fallback-degradation (reliability)

The network mitigates transient congestion by detecting buffer contention and PFCing back into the network.

Tradeoffs

Cluster scaling approach

Chose multi-building regional cluster over single-building 24K cluster.

Rationale: Next-generation generative AI models demanded more than 24K GPUs.

Congestion mitigation

Chose deep buffers plus aggressive PFC threshold tuning over allowing PFC/head-of-line blocking to propagate.

Rationale: The design absorbs congestion locally and reduces performance-damaging queue buildup.

Fleet rollout

Chose one-DC monitored rollout over immediate fleet-wide network fix.

Rationale: The staged rollout avoided downtime and checked for side effects while jobs were ramping.

Gotchas (what bit them)

PFC can create head-of-line blocking at AI scale.

Root cause: Lossless traffic and bursty AI workloads create congestion and queue buildup.

Resolution: Meta installed deep-buffer CTSWs and tuned RTSW thresholds to reduce PFC propagation.

Tiny lossy-queue traffic caused workload degradation.

Root cause: RoCE control packets were carried in a lossy queue that dropped under ramp-up, even though it was less than 1% of traffic.

Resolution: Meta moved RoCE control traffic back into the lossless queue and retuned buffers.

Some performance bugs only reproduce at large GPU scale.

Root cause: The firmware regression required an 8K GPU job across multiple DCs to reproduce and normal metrics did not point to the issue.

Resolution: Meta traced the problem to divergent NIC firmware settings and fixed part of the lost performance.

Multi-building AI clusters create unavoidable distance penalties.

Root cause: The 100K+ cluster spans five data centers over 3 km, adding propagation delay.

Resolution: Meta developed low-latency networking and invested in monitoring to preserve performance.

Evaluation

Meta evaluates the network through GPU scale, cluster topology, health dashboards at all layers, congestion/drop behavior, workload performance, fleet rollout monitoring, and large-scale reproduction tests.

Metrics: 24K GPUs · 100K+ GPUs · 38 AI zones · Five buildings · 20 data halls · More than 3 km span · Less than 1% lossy queue traffic · 8K GPU reproduction job · 30% performance degradation · 10% performance recovered from firmware fix

Outcomes

• Meta built a 24K GPU cluster for Llama 3 in 2023.
• Meta evolved to a 100K+ GPU, multi-building regional cluster.
• Deep-buffer switches and PFC threshold tuning mitigated expected congestion.
• RoCE control traffic was moved to the lossless queue to avoid workload degradation.
• Transient congestion fixes were rolled out fleet-wide without downtime.
• A divergent NIC firmware issue was identified and partially recovered performance.
• Meta expects Prometheus to span an entire metropolitan area and possibly cross a million GPUs.

The case reports a 24K GPU Llama 3 cluster, a 100K+ GPU cluster with five buildings, 20 data halls, 38 AI zones, more than 3 km span, an 8K GPU reproduction job, 30% DC-level degradation, about 10% recovered performance, and future clusters that may cross one million GPUs over distances above 100 km.