Scaling LLM Training and Inference with FP8 Precision

deepl · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM training and inference efficiency

DeepL needed to scale larger language AI models while preserving low-latency production inference and high translation quality.

Constraints:

Moving from BF16 to FP8 reduces precision and range, so training quality had to remain comparable.
Production inference must balance throughput with latency because customer experience depends on response speed.
The system had to exploit H100 native FP8 support across both training and inference.

Architecture

DeepL moved training and inference from BF16 toward FP8 on a 544-H100 DGX SuperPOD. Training uses NVIDIA Transformer Engine with E4M3 in forward pass, E5M2 in backward pass, scaling factors, and optimization work with NVIDIA. Inference uses TensorRT-LLM to build optimized engines with kernel fusion, optimized CUDA/C++ code, KV caching, and continuous in-flight batching.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["DGX SuperPOD H100 cluster"] --> B["Transformer Engine training"]
  B --> C["FP8 scaling factors"]
  B --> D["Training pipeline"]
  D --> E["Model quality evaluation"]
  A --> F["TensorRT-LLM inference"]
  E --> F
  F --> G["Production translation service"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`DGX SuperPOD H100 cluster`	544-GPU compute substrate with native FP8 tensor core support.	`Transformer Engine training`, `TensorRT-LLM inference`
`Transformer Engine training`	Mixed-precision training library that manages FP8/BF16 conversion and scaling factors.	`DGX SuperPOD H100 cluster`, `FP8 scaling factors`, `Training pipeline`
`FP8 scaling factors`	Additional scalars used to avoid overflow/underflow and preserve useful numeric range.	`Transformer Engine training`
`Training pipeline`	Pre-training, fine-tuning, distillation, reinforcement learning, and parallelization strategies.	`Transformer Engine training`, `Model quality evaluation`
`Model quality evaluation`	Training loss, validation perplexity, and language-expert quality assessment.	`Training pipeline`, `TensorRT-LLM inference`
`TensorRT-LLM inference`	Production inference engine builder with FP8, kernel fusion, CUDA/C++ optimizations, KV cache, and batching.	`DGX SuperPOD H100 cluster`, `Model quality evaluation`, `Production translation service`
`Production translation service`	User-facing DeepL language AI features served under latency constraints.	`TensorRT-LLM inference`

Data flow

During training, model tensors are represented in FP8 with scaling factors and Transformer Engine manages conversion between FP8 and BF16. After pre-training, fine-tuning, distillation, and RL, TensorRT-LLM builds inference engines from the weights and optimizes production serving with FP8 kernels, KV caching, and batching to double throughput within the same latency envelope.

3 supporting quotes

“We transitioned our existing training code from BF16 to FP8 using NVIDIA Transformer Engine”article, stated

“Transformer Engine provides essential components that facilitate mixed-precision training, seamlessly managing the conversion between FP8 and BF16 formats”article, stated

“NVIDIA TensorRT-LLM, which is NVIDIA’s solution for scalable LLM inference, and which supports FP8.”article, stated

Why this architecture

FP8 lets DeepL increase training and inference throughput with the same hardware and latency budget, but only if scaling, precision format choices, and quality checks keep the lower-precision math stable.

3 supporting quotes

“By computing matrix multiplications in FP8, we can increase throughput when training and deploying our Large Language Models”article, stated

“This means translations that outperform our previous models by 1.4x for European languages and 1.7x for more complex language pairs”article, stated

“FP8 has effectively doubled the effective capacity of our LLMs in terms of throughput.”article, stated

Models used: DeepL next-generation LLMs · 1.5B comparison model · NVIDIA Transformer Engine · NVIDIA TensorRT-LLM

Scale

DeepL deployed a DGX SuperPOD with 544 NVIDIA H100 GPUs.
“with 544 NVIDIA H100 Tensor Core GPUs”article, stated
FP8 increased MFU from 44.6% to 67%, accelerating training by 50%.
“increased from 44.6% MFU to 67% MFU with FP8, effectively accelerating our model training by 50%.”article, stated
Another training setup improved performance by 25% over 15 months to 80% MFU.
“incrementally increase training performance by 25% over the course of 15 months, taking us up to 80% MFU.”article, stated
DeepL trained a 1.5B model on three trillion tokens for FP8/BF16 quality comparison.
“We trained a 1.5B model on three trillion tokens and then compared the quality of the FP8 training vs BF16.”article, stated
FP8 can handle double the throughput for the same latency as BF16 for most batch sizes.
“for most batch sizes, FP8 can handle double the throughput for the same degree of latency as BF16.”article, stated

Alternatives considered (roads not taken)

Continue BF16-only training and inference

Rejected because: FP8 unlocked higher training throughput and doubled inference capacity within the same latency constraints.

“FP8 enables faster computation and requires significantly reduced memory compared to BF16.”article, stated

Use one FP8 format for all phases

Rejected because: DeepL used E4M3 for forward precision and E5M2 for backward gradient range.

“using E4M3 in the forward pass for training, and then using E5M2 in the backward pass.”article, stated

Optimize throughput without latency constraints

Rejected because: Batching improves throughput but can harm customer experience through higher latency.

“Batching multiple requests enables greater throughput, but at the cost of increased latency for each individual request.”article, stated

Patterns

`latency-optimization` (production-operations)

FP8 inference doubles throughput at comparable latency, expanding production capacity without degrading user experience.

“while maintaining comparable training quality to BF16 precision.”summary, stated

`cost-aware-routing` (infrastructure)

Lower precision improves hardware utilization and memory efficiency, enabling larger models and more requests on the same infrastructure.

“On the flip side to this lack of range and precision, FP8 enables faster computation and requires significantly reduced memory compared to BF16. This is hugely valuable for inference, and with the right approach, it can be hugely valuable for accelerating training as well.”article, stated

`fine-tuning-adaptation` (model-training)

DeepL's model lifecycle includes pre-training, fine-tuning, distillation, reinforcement learning, and deployment under FP8-aware training/inference.

“After pre-training, we fine-tune our models on certain tasks, distil large models into smaller models, do reinforcement learning”article, stated

Tradeoffs

Numeric precision

Chose FP8 mixed precision over BF16 everywhere.

Rationale: FP8 reduced memory and improved throughput while maintaining comparable quality in DeepL's tests.

“The net result of moving from BF16 to FP8 is that we are able to train our models faster, with reduced demand on memory”article, stated

FP8 format assignment

Chose E4M3 forward and E5M2 backward over one uniform FP8 representation.

Rationale: Forward pass benefits from precision; backward pass benefits from range for gradients.

“format with higher precision for predicting the probability distribution of the next token, and then the format with lower precision, but a higher range, to compute the gradients”article, stated

Gotchas (what bit them)

FP8 has lower precision and range than BF16.

Root cause: FP8 uses half as many bits, limiting exponent and mantissa representation.

Resolution: Use E4M3/E5M2 formats where each fits best and store scaling factors with FP8 tensors.

“Because of this, it has a smaller range and lower precision than BF16. It can represent a narrower range and fewer numbers within that range.”article, stated

Low-precision training can underflow or overflow.

Root cause: FP8 has limited numeric range during tensor operations.

Resolution: Store additional scaling factors and account for them in calculations.

“This involves storing additional scaling factors alongside the FP8 weight tensors to overcome the limited range and prevent overflow and underflow.”article, stated

Inference throughput improvements can harm latency.

Root cause: Batching more requests increases throughput but also increases per-request latency.

Resolution: Evaluate FP8/BF16 at fixed latency budgets and use TensorRT-LLM optimizations.

“It's inevitable that, as throughput increases, latency tends to increase as well. Batching multiple requests enables greater throughput, but at the cost of increased latency for each individual request.”article, stated

Evaluation

DeepL evaluates FP8 through MFU, training loss, validation perplexity, downstream language quality judged by experts, latency/throughput curves, and production request capacity.

Metrics: Model FLOPS Utilization · training loss · validation perplexity · throughput · latency · translation quality multiplier

“we tested training one of our models in both formats. This enabled us to compare training losses and downstream quality.”article, stated

“In this case, we tested how the model got on when working with English and German. We compared validation perplexity, which quantifies the uncertainty a model experiences when predicting the next token in a sequence.”article, stated

Outcomes

FP8 improved MFU from 44.6% to 67% and accelerated training by 50%.
DeepL observed no downstream quality degradation in an English/German validation-perplexity test.
FP8 doubled effective inference capacity at the same latency for most batch sizes.
Next-generation translations improved 1.4x for European languages and 1.7x for English-Japanese style complex pairs.

The article is a company-authored performance report, but it provides concrete training and inference metrics.

“In this practical scenario, we actually found no degradation in quality with FP8 training compared to BF16.”article, stated

“It’s also ensured that we are able to apply those LLMs effectively, to deliver optimum customer experiences and scale the impact of our Language AI in the wild.”article, stated