High-Performance AI Network Infrastructure for Distributed Training at Scale

meta · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: distributed AI training network infrastructure

Meta needed distributed AI training jobs to move checkpoint and training data across rapidly evolving multi-NIC GPU hosts without bottlenecking on a single interface, leaving GPUs idle, or requiring model code to understand the underlying hardware topology.

Constraints:

• checkpoint data grew from hundreds of gigabytes to tens of terabits and even petabits in the source's framing
• distributed training jobs fail due to host or GPU problems and require fast checkpoint-based recovery
• bootstrapping thousands of training nodes creates connection bursts and huge data movement
• single-NIC binding funnels checkpoint and ingestion traffic through one interface while other NICs are underutilized
• model code should remain hardware-agnostic across development boxes and production fleets
• solutions must use all available NIC resources at line rate without requiring fabric-switch surgery
• NUMA cross-socket communication can add latency and reduce throughput

Architecture

Meta built a bidirectional multi-NIC utilization design for AI training hosts. For egress, each container exposes the host's physical NICs through lightweight netkit devices and assigns IPs to every NIC plus a central task IP; applications can bind explicitly, or hardware-agnostic applications use an in-container ECMP setup that spreads outgoing flows across equal-cost NIC paths, with eBPF hooks driving packets directly to physical NICs at line rate. For ingress, a VIP injector runs inside the container, treats the task IP as a virtual IP, establishes a BGP session with the rack switch, and announces multiple equal-cost next hops corresponding to the host NICs. The rack switch then uses consistent hashing to spread inbound flows and automatically removes failed links from the ECMP group. The design preserves route aggregation beyond the rack, adds security checks so only the training job can announce its task IP, and is being advanced with NUMA-aware egress and hardware RX flow steering for latency improvement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Distributed training job"] --> B["Training container"]
  B --> C["Netkit NIC exposure"]
  C --> D["Physical NIC pool"]
  B --> E["ECMP egress multipath"]
  E --> F["eBPF forwarding hook"]
  F --> D
  B --> G["VIP injector"]
  G --> H["Rack switch ECMP"]
  H --> D
  H --> I["Task IP route aggregation"]
  G --> J["Task IP security check"]
  D --> K["NUMA-aware steering"]

Components

Data flow

A distributed training job runs inside a container with access to multiple host NICs. Checkpoint, data-ingestion, and hot-sparing traffic can leave the container through ECMP equal-cost paths across all exposed physical NICs, with flow stickiness to avoid packet reordering and eBPF/netkit support for line-rate forwarding. Inbound traffic targets the task IP announced by a VIP injector over BGP to the rack switch; the switch installs ECMP next hops for the host NICs, hashes flows across them, removes failed next hops automatically, and prevents task-IP route leakage beyond the rack. This bidirectional spread turns a single overloaded NIC path into multiple lanes for checkpoint loading, job reads, and training data movement.

Why this architecture

Meta chose host-level ECMP and BGP/VIP injection because the bottleneck had shifted from compute to bandwidth, and GPU clusters were leaving costly network capacity idle. The design uses all NICs without changing model code, spreads both egress and ingress traffic, preserves fabric-level scalability through rack-local route aggregation, and lets rack switches adapt to link failures. The same architecture also leaves room for NUMA-aware steering where latency is still affected by cross-socket traffic.

Models used: distributed AI training jobs · training models

Scale

• Checkpoint data grew by 10x from hundreds of gigabytes to tens of terabits and even petabit scale in the talk's wording.
  “Last year itself we have seen checkpoint data where at hundreds of gigs and this year we could see that it is like 10x with tens of terabits”transcript, stated
• Training bootstrapping involves thousands of execution nodes downloading parameters and distributing weights.
  “the thousands of nodes the training model training execution nodes will be trying to download the parameters will try to distribute the weights”transcript, stated
• Hardware evolved from two NICs to four 100G NICs and then four 200G ports, yielding an 800G host funnel.
  “people have migrated from two nicks to the four four nicks with the 100 gig.”transcript, stated

  “4 nicks with 200 gig port speed is going to give us the 800 gig funnel.”transcript, stated
• A real training job's read latency spiked from 8 seconds to 300 seconds before the fix.
  “The job read latency spiked from 8 seconds to 300 seconds.”transcript, stated
• The bidirectional solution reduced job read latency from 300 seconds to 1 second.
  “we see a substantial reduction from 300 second to 1 second.”transcript, stated
• Checkpoint loading latency dropped from 800 seconds to 100 seconds.
  “the check point loading latency also dropped from 800 seconds to 100 second.”transcript, stated
• A four-NIC stress test distributed traffic evenly and improved throughput by four times.
  “each of the nick take about one quarter of the stress load. This means we in improve the throughput by four times.”transcript, stated

Alternatives considered (roads not taken)

Leave one NIC as a failure link

Rejected because: Checkpoint and training data movement grew enough that spare NIC capacity needed to be used for data transfer, not left idle only for failure scenarios.

Require training applications to bind to specific NICs

Rejected because: That makes model code aware of underlying hardware, while Meta wanted the same model to run across development boxes and production hardware without changes.

Modify the fabric switches deeply

Rejected because: The solution requirement was to utilize host-level hardware correctly without doing fabric-switch surgery.

Use rack-switch ECMP alone without task-IP route control

Rejected because: Announcing every task to the network could overload routing unless task addresses are aggregated and route propagation is stopped at the rack.

Patterns

latency-optimization (infrastructure)

Meta spread checkpoint and data-ingestion traffic across all available NICs, added line-rate ECMP/eBPF forwarding, and planned NUMA-aware steering to reduce read and checkpoint latency.

durable-execution (reliability)

The network design protects distributed training recovery by accelerating checkpoint reads and loading, supporting hot sparing, and reducing fallback to stale checkpoint data.

fallback-degradation (reliability)

Rack-switch ECMP removes failed physical NIC next hops and rehashes affected flows across remaining NICs, keeping inbound traffic balanced without packet loss.

cost-aware-routing (operations)

The routing work was motivated by avoiding GPU idle time from network bottlenecks, because idle GPUs waste money during checkpoint and data-loading stalls.

safety-policy-enforcement (network security)

Task-IP announcement is guarded so only the owning training job can announce the task IP, preventing hijacking of training traffic.

Tradeoffs

Keep model applications hardware-agnostic

Chose container-level ECMP and VIP injection over requiring model code to bind to specific NICs.

Rationale: The same training model should run from development boxes to production fleets even as hardware evolves from one to two to four high-speed NICs.

Use ECMP for egress from the container

Chose equal-cost multipath across all available NICs over single default NIC selection.

Rationale: ECMP lets hardware-agnostic applications distribute outgoing checkpoint and data traffic across NICs at line rate while preserving flow stickiness.

Use BGP/VIP injection for ingress

Chose task IP announced as a virtual IP with multiple equal-cost next hops over inbound traffic arriving through a single last-hop NIC.

Rationale: Inbound flows need the same multi-lane treatment as outgoing traffic, but the rack switch needs task-IP next-hop information to distribute them.

Localize routing changes to the rack

Chose rack-local task-IP announcements with aggregation and propagation stopping over leaking per-task routes into the wider network.

Rationale: The ingress design scales because it adds no actual routes beyond the rack while still allowing rack-switch ECMP to balance flows across host NICs.

Gotchas (what bit them)

Single-NIC defaults create a hidden bottleneck on multi-NIC GPU hosts.

Root cause: Without special NIC selection, training jobs bind to the first available NIC, so checkpoint and ingestion traffic overload that interface while other NICs sit underutilized.

Resolution: Expose all NICs to the container and provide ECMP-based egress load balancing for hardware-agnostic applications.

Checkpoint delays damage hot-sparing quality.

Root cause: Hot sparing depends on current checkpoint updates; if checkpoint transfer is slow, standby GPU servers fall back to older state.

Resolution: Improve checkpoint network throughput with bidirectional multi-NIC traffic spreading.

Host-level ECMP can create routing-scale risk if every task route leaks beyond the rack.

Root cause: Large numbers of simultaneous training tasks could overload routing tables if task IPs are globally propagated.

Resolution: Use hierarchical address allocation and rack-switch policy to aggregate task IP routes and stop propagation beyond the rack.

NUMA-unaware NIC selection can erase part of the latency win.

Root cause: If packets are routed through a NIC attached to the wrong NUMA domain, cross-socket memory reads add latency and drop throughput.

Resolution: Add transparent NUMA awareness and hardware RX flow steering so packets land near the CPU socket where the flow exists.

Evaluation

Meta evaluated the bidirectional multi-NIC design with a real incident where job read latency spiked, checkpoint-loading measurements, and a four-NIC GPU-server stress test that compared traffic distribution and throughput before and after the solution.

Metrics: job read latency reduced from 300 seconds to 1 second · checkpoint loading latency reduced from 800 seconds to 100 seconds · four-NIC stress test distributed about one quarter of load to each NIC · throughput improved by four times · no actual route introduced beyond the rack · traffic balanced across remaining NICs without packet loss after link failure

Outcomes

• Job read latency dropped from 300 seconds to 1 second after applying the bidirectional solution.
• Checkpoint loading latency dropped from 800 seconds to 100 seconds.
• A four-NIC stress test distributed traffic across all four NICs and improved throughput by four times.
• The solution has been applied to various training models.
• The design preserved application transparency for both egress and ingress paths.

The solution targets distributed AI training jobs across racks, clusters, data centers, regions, and hybrid cloud settings, with hosts evolving from two NICs to four 100G NICs and four 200G NICs, thousands of training nodes, and checkpoint data growth described from hundreds of gigabytes to tens of terabits and beyond.