Scaling AI Infrastructure: Network Architecture and Communication Optimization at Microsoft

meta · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI training network architecture and communication optimization

Microsoft Azure needed network designs, routing, validation, and communication-library tuning that could support rapidly scaling AI training clusters without wasting capacity or hiding bad nodes, links, or NICs.

Constraints:

Architecture

The talk lays out an Azure AI network stack that separates topology selection, routing, cluster validation, communication libraries, and reliability. Microsoft uses rail-optimized non-oversubscribed public clusters for generality; multi-pod and rail-only dedicated AI topologies for cost and workload-specific efficiency; smart NIC/simple switch/smart CCL routing ideas; SuperBench and IB Pulse for validation; and TACCL/MSCCL-style communication algorithms for hierarchical collectives, cluster-specific tuning, and non-RTT-based communication.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Azure GPU clusters"] --> B["Network topology layer"]
  B --> C["Routing control layer"]
  C --> D["Communication libraries"]
  D --> A
  A --> E["Validation frameworks"]
  E --> F["SuperBench"]
  E --> G["IB Pulse"]
  H["Networking partners"] --> E
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Azure GPU clusters AI training clusters based on A100/H100 SKUs and other Azure GPU infrastructure. Network topology layer, Communication libraries, Validation frameworks
Network topology layer Rail-optimized, multi-pod, rail-only, and non-oversubscribed topology choices for public or dedicated AI clusters. Routing control layer, Communication libraries
Routing control layer Smart NIC, smart switch, and smart CCL routing strategies that adapt network paths to workload feedback. Network topology layer, Communication libraries
Communication libraries Collective communication layer including TACCL/MSCCL-style algorithms and cluster-specific tuning. Azure GPU clusters, Routing control layer
SuperBench Node and cluster benchmark framework for classifying good and bad nodes and targeted benchmarks. Validation frameworks, Azure GPU clusters
IB Pulse In-house benchmarking framework for network validation and topology-area testing. Validation frameworks
Validation frameworks Cluster bring-up and health tooling that finds bad nodes, links, and NICs before workload impact. SuperBench, IB Pulse, Network topology layer
Networking partners Hardware and tooling partners helping test links before full-fabric integration. Validation frameworks

Data flow

Training workloads run on Azure GPU clusters whose topology determines communication paths. Validation tools inspect nodes, network regions, links, and NICs before and during cluster bring-up. Communication libraries receive topology/routing hints or provide hints to the network, then choose collective algorithms and tuning for the specific hardware, scale, and message size.

3 supporting quotes
“I Al also want to talk about the validation Frameworks and uh com library and reliability”transcript, stated
“the CCL or the communication Library can have uh can have a say on like the the routing configuration”transcript, stated
“the super bench monitors the Behavior Uh based on the results right and then based on those right I mean it can model and come up with the different set of benchmarks”transcript, stated

Why this architecture

The architecture is layered because topology, routing, validation, and collectives interact. Public clusters need generic full-bandwidth communication, but dedicated AI clusters can save switch cost by matching topology and collective algorithms to workload patterns. Validation frameworks reduce cluster bring-up time by finding hardware and network defects before training runs consume capacity.

3 supporting quotes
“clusters where uh I mean you have to support a variety of different workloads”transcript, stated
“for dedicated uh AI clusters right I mean you can actually you have the opportunity to fine tune or design it based on the workload characteristics”transcript, stated
“reducing the time um for um U like while bringing up the cluster right so that's the main uh main goal”transcript, stated

Models used: large AI training workloads · H100 clusters · A100 clusters

Scale

Alternatives considered (roads not taken)

Rail-optimized non-oversubscribed public cluster

Rejected because: It is appropriate for broad workloads but not always the cheapest or most tuned choice for dedicated AI clusters.

“any node can talk to anybody else in the cluster without any performance impact”transcript, stated

Top-layer switches in rail-only dedicated clusters

Rejected because: Some targeted workloads can keep communication within rails and save cost without the top layer.

“you don't now you don't need the top layer of switches this may not work for all workloads but at least uh uh for some uh targeted workloads”transcript, stated

RTT-based algorithms for region-scale communication

Rejected because: Region-scale cluster communication needs non-RTT-based algorithms.

“when we are scaling out we have the need to um communicate between regions of uh clusters right um the rtt based algorithms will not work”transcript, stated

Network routing independent of communication libraries

Rejected because: The talk argues the CCL can inform routing and topology choices more gracefully.

“the CCL or the communication Library can have uh can have a say on like the the routing configuration”transcript, stated

Patterns

latency-optimization (infrastructure)

Microsoft tunes topology, routing, and collectives to reduce communication overhead and improve network utilization.

“you get a better Network utilization and you you actually are coaling the data so you get better performance also”transcript, stated

cost-aware-routing (infrastructure)

Rail-only topology and communication algorithms reduce switch layers for targeted workloads.

“you can actually save a lot of U money right I mean this translates to a lot of cost savings”transcript, stated
“Translating that to a lot of uh cost savings right because you don't now you don't need the top layer of switches”transcript, stated

eval-driven-development (evaluation)

SuperBench and IB Pulse benchmark nodes, network areas, links, and NICs to validate cluster health.

“we have uh our um answer here is super bench and also our in-house uh uh benchmarking framework called IB pulse”transcript, stated
“The Benchmark tool will output a list of say good nodes or uh or bad nodes or like bad bad bad links or even Nicks”transcript, stated

structured-output-contract (engineering-practice)

Communication-library and routing designs use topology hints and cluster-specific configurations rather than treating all clusters the same.

“running on aure ND V4 and like this particular scale and this message size give me the best uh tuning for that”transcript, stated

Tradeoffs

Public cluster topology

Chose rail-optimized non-oversubscribed design over more workload-specific reduced topologies.

Rationale: Public clusters need to support many communication patterns without performance impact.

“any node can talk to anybody else in the cluster without any performance impact”transcript, stated

Dedicated AI topology

Chose rail-only or multi-pod designs where workload permits over always using full top-layer switching.

Rationale: Targeted workloads can save cost by keeping communication within rails.

“this translates to a lot of cost savings right so you are uh taking the the top layer of switches out”transcript, stated

Routing intelligence

Chose smart CCL / smart NIC feedback over simple fixed routing only.

Rationale: The communication layer can provide workload hints that make routing more graceful.

“Smart Switch based designs can handle this well but um the Nick based or the feedback for the Nick right or the feedback from the communication Library”transcript, stated

Gotchas (what bit them)

Topology choices that work for public clusters can waste money for dedicated AI clusters.

Root cause: Public clusters need all-to-all flexibility, while dedicated workloads may have predictable collective patterns.

Resolution: Microsoft considers multi-pod and rail-only topology for workload-specific clusters.

“you don't have to uh I mean Focus only on say all reduce or um like reduce scatter like you can support all to all all all the different models”transcript, stated
“rail only uh topology right so um that means that um you are enabling communication only within that rail”transcript, stated

Communication libraries must understand topology.

Root cause: Collective performance depends on routing, rail boundaries, message size, and cluster layout.

Resolution: Microsoft explores smart CCL routing feedback and cluster-specific MSCCL/TACCL tuning.

“you can also U get the uh hints from the communication library to the network”transcript, stated
“you have like really tuned uh set of configurations for different clusters”transcript, stated

Bad links and NICs can hide during cluster bring-up.

Root cause: Large fabrics require targeted testing below specific topology levels rather than only whole-cluster tests.

Resolution: SuperBench and IB Pulse output good/bad nodes, links, and NICs based on expected bandwidth thresholds.

“you can have estimate of like what's the expected bandwidth for this Benchmark that I should see on on a particular link”transcript, stated
“if it's less than a particular threshold yeah you can always list as uh bad bad links Etc”transcript, stated

Evaluation

The talk evaluates infrastructure through benchmark rankings, topology bandwidth distributions, SuperBench node classification, IB Pulse network testing, expected bandwidth thresholds, and communication-library performance tuning.

Metrics: #1 cloud Top500, rank 3 overall · #1 cloud MLPerf, rank 2 overall · 1K/2K GPU papers versus multiple hundreds of thousands of GPUs · Expected bandwidth thresholds for links · Good/bad node, link, and NIC lists · Cluster-specific scale and message-size tuning

“for the top 500 we made number one in in the cloud uh with a rank of three overall”transcript, stated
“The Benchmark tool will output a list of say good nodes or uh or bad nodes or like bad bad bad links or even Nicks”transcript, stated

Outcomes

The talk references Top500 rank 3 overall, MLPerf rank 2 overall, Azure A100/H100 SKUs, 1K/2K GPU clusters as formerly large, and current industry movement toward multiple hundreds of thousands of GPUs.

“we made number one in in the cloud uh with a rank of three overall and also number one in Cloud for ML perf and rank two overall”transcript, stated
“those uh new strategies and another main thing is that we I mean the the need for fine grained uh routing that's becoming more uh more and more”transcript, stated