Meta's Hardware Reliability Framework for AI Training and Inference at Scale

meta · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI hardware reliability for training and inference

Meta's large-scale AI training and inference infrastructure is vulnerable to hardware faults, especially silent data corruptions, that can halt training, create illusory progress, corrupt inference outputs, and waste massive compute resources.

Constraints:

• Training involves thousands of accelerators in synchronous clusters where one component failure can interrupt or halt work.
• Silent data corruptions can miscompute without leaving detectable traces.
• Accelerators have higher complexity and limited telemetry compared with CPUs.
• SDC manifestations can depend on data, voltage, frequency, temperature, and device lifecycle.
• Inference corruptions can affect thousands of consumers and bypass normal privacy or integrity policies.

Architecture

Meta uses a layered hardware-reliability framework for AI training and inference. Fleet-level mechanisms include Fleetscanner targeted micro-benchmarks, Ripple co-located millisecond/second tests, and Hardware Sentinel kernel-space anomaly analytics. Training mitigation combines reductive triage, deterministic training, and hyper-checkpointing with stack-level approaches such as gradient clipping, algorithmic fault tolerance, tri-variate shadow-node training, parameter vulnerability factors, and divergence detection. A factory-to-fleet reliability strategy feeds lessons back into MTIA design, telemetry, diagnostics, firmware hooks, and stack resilience.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Workloads["AI training and inference workloads"] --> Infra["Global AI infrastructure"]
  Workloads --> Stack["Stack-level resilience"]
  Infra --> Fleet["Fleetscanner"]
  Infra --> Ripple["Ripple"]
  Infra --> Sentinel["Hardware Sentinel"]
  Infra --> Triage["Reductive triage"]
  Infra --> Checkpoint["Hyper-checkpointing"]
  Triage --> Checkpoint
  Factory["Factory-to-fleet feedback"] --> Infra
  Factory --> Sentinel
  Factory --> Fleet
  Factory --> Ripple

Components

Data flow

AI training and inference workloads run across Meta's global compute, storage, network, and accelerator infrastructure. Hardware telemetry, micro-benchmarks, co-located tests, kernel exceptions, anomaly detectors, and workload signals identify static, transient, and silent faults. Suspect devices are triaged, quarantined, repaired, or mapped differently, while training resumes from checkpoints or uses stack-level fault tolerance to reduce wasted computation.

Why this architecture

Meta uses multiple detection and mitigation layers because no single method covers the SDC space. Fleetscanner provides periodic directed tests, Ripple accelerates fleet coverage while co-locating with workloads, Hardware Sentinel avoids dedicated test allocation, and stack-level resilience handles corruption that slips past infrastructure diagnostics. Factory-to-fleet feedback moves reliability earlier into silicon and telemetry design.

Models used: Llama 3 · PyTorch training workloads · large-scale AI models · text-to-image generation models · object segmentation models

Scale

• Meta's AI infrastructure spans globally distributed data centers.
  “connected via network fabric across globally distributed data centers.”article, stated
• Large-scale training involves thousands of accelerators.
  “Training large-scale models involves thousands of accelerators in a synchronous environment, where any component failure can interrupt or halt the process.”article, stated
• Hardware failures accounted for over 66% of training interruptions in Llama 3 experience.
  “with over 66% of training interruptions due to such failures.”article, stated
• Modern accelerator SDCs occur at about one fault per thousand devices.
  “silent data corruptions now occur at about one fault per thousand devices, much higher than cosmic-ray-induced soft errors.”article, stated
• Fleetscanner covers the whole fleet every 45 to 60 days.
  “Tests are scheduled periodically, covering the entire fleet every 45 to 60 days.”article, stated
• Hardware Sentinel outperforms testing-based methods by 41%.
  “Hardware Sentinel outperforms testing-based methods by 41% across architectures, applications, and data centers.”article, stated

Alternatives considered (roads not taken)

Rely on simple health checks and visible failures

Rejected because: Static failures are easy to detect, but SDCs miscompute without traces and can remain unnoticed.

Assume AI training is naturally resilient to SDCs

Rejected because: Meta found self-resilience applies only to limited SDC manifestations and is inadequate in realistic scenarios.

Use only testing-based SDC detection

Rejected because: Testing can be too slow or instruction-specific, so Meta adds analytical and workload-co-located methods.

Solve reliability only in silicon or only in software

Rejected because: Meta argues resilience must span factory-to-fleet silicon lifecycle and stack-level software/application techniques.

Patterns

observability-tracing (operations)

Meta uses telemetry, kernel-space exception analysis, anomaly detection, fleet signatures, and diagnostic hooks to trace hardware faults back to devices.

eval-driven-development (evaluation)

Reliability is verified through targeted micro-benchmarks, deterministic training, co-located tests, performance testing, coverage maps, and lifecycle diagnostics.

This approach involves running a known effective model for a few training iterations to ensure there are no NaNs or gradient divergences."

— article, stated

durable-execution (runtime)

Hyper-checkpointing and checkpoint restore keep long-running synchronous training from losing all progress when faulty nodes are isolated.

This method involves creating checkpoints at increasingly high frequencies to facilitate faster identification and isolation of the corrupting node."

— article, stated

safety-policy-enforcement (control)

Reliability controls prevent inference corruptions from bypassing privacy, safety, or content-integrity policies.

Tradeoffs

Use multiple SDC detectors

Chose Fleetscanner, Ripple, and Hardware Sentinel together over One periodic benchmark-only detector.

Rationale: Fleetscanner gives directed fleet tests, Ripple gives faster co-located coverage, and Hardware Sentinel provides test-agnostic analytics.

Checkpoint aggressively during suspected corruption

Chose Hyper-checkpointing over Sparse checkpointing with broad restart windows.

Rationale: Higher checkpoint frequency helps isolate corrupting nodes faster while preserving training throughput.

Use algorithmic fault tolerance

Chose Fault tolerance built into training algorithms over Always detecting and triaging every corruption after the fact.

Rationale: Algorithmic tolerance can handle a range of corruptions and reduce detection/triage needs, but adds engineering and overhead.

Map resilience by layer vulnerability

Chose Parameter vulnerability factors over Protecting every layer uniformly.

Rationale: PVF directs resilient hardware to vulnerable layers while accepting cost, area, power, or performance tradeoffs only where useful.

Gotchas (what bit them)

Silent data corruption can create an illusion of training progress.

Root cause: Corrupted gradient variance stays within numeric bounds and is treated as legitimate training data.

Resolution: Use divergence detection, reductive triage, checkpointing, gradient clipping, and algorithmic fault tolerance to detect or contain corruption.

NaN propagation from one accelerator can halt an entire cluster.

Root cause: A few computations on one accelerator can create NaNs that propagate through iteration, accelerator, host, and cluster domains.

Resolution: Identify and quarantine offending accelerators/nodes, then resume with healthy nodes from checkpoints.

SDCs can take weeks or months to become observable.

Root cause: Subtle numeric corruptions do not trigger NaN traps and can aggregate slowly.

Resolution: Use periodic/frequent testing, anomaly detection, and workload-aware detection throughout the device lifecycle.

Inference SDCs can silently corrupt production decisions.

Root cause: Persistent hardware miscomputations can alter recommendation or LLM outputs without normal constraint boundaries catching them.

Resolution: Apply telemetry, divergence detection, anomaly detectors, and device isolation before inference clusters continue serving.

Evaluation

Meta evaluates hardware reliability through fleet scans, co-located tests, kernel-space exception analytics, deterministic training, binary-search triage, divergence maps, ServiceLab performance tests, and factory-to-fleet telemetry feedback.

Metrics: over 66% of training interruptions tied to hardware failures · one SDC fault per thousand accelerator devices · Fleetscanner full-fleet coverage every 45-60 days · Ripple fleet-wide coverage in days · Hardware Sentinel 41% improvement over testing-based methods · weeks or months to observe subtle SDC effects

Outcomes

• Meta productionized SDC detection across AI and non-AI infrastructure.
• Fleetscanner, Ripple, and Hardware Sentinel provide complementary in-fleet coverage for silent data corruption.
• Training mitigations include reductive triage, deterministic training, hyper-checkpointing, gradient clipping, algorithmic fault tolerance, and shadow-node strategies.
• Inference mitigations include PVF and divergence detection for corruptions that affect outputs at scale.
• Meta moved SDC lessons into MTIA and factory-to-fleet reliability practices.

The case cites global data centers, thousands of accelerators, over 66% of Llama 3 training interruptions due to hardware failures, SDC rates near one per thousand accelerator devices, 45-60 day Fleetscanner coverage, and 41% Hardware Sentinel improvement.