CPU-Based Deployment of Large MoE Models Using Intel Xeon 6 Processors

lmsys · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: CPU serving for large MoE language models

LMSYS/SGLang and the Intel PyTorch Team needed to make very large MoE models such as DeepSeek R1 deployable without the usual fleet of expensive accelerators, while preserving useful inference latency and throughput on Xeon CPU servers.

Constraints:

DeepSeek R1-class models have massive memory requirements.
The target was CPU-only serving on a single Xeon 6 node.
MoE and attention kernels are bandwidth- and parallelism-sensitive.
x86 CPUs lack native FP8 support.
NUMA topology can cause remote-memory overhead if tensor parallel work is not placed carefully.

Architecture

The team added a native SGLang CPU backend for Intel Xeon 6 with AMX, mapping attention, MLA, MoE, quantization, and FP8 kernels to CPU primitives. The serving path combines extend/decode attention kernels, MLA packing/head-folding, optimized INT8/FP8 MoE kernels, and tensor-parallel multi-NUMA placement so DeepSeek R1 and other large models can run on CPU nodes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  sglang-cpu-backend["SGLang CPU backend"] --> extend-attention-kernel["Extend attention kernel"]
  sglang-cpu-backend --> decode-attention-kernel["Decode attention kernel"]
  sglang-cpu-backend --> mla-optimization-layer["MLA optimization layer"]
  sglang-cpu-backend --> moe-kernel["MoE kernel"]
  sglang-cpu-backend --> fp8-gemm-path["FP8 GEMM path"]
  sglang-cpu-backend --> multi-numa-tensor-parallel-runtime["Multi-NUMA tensor parallel runtime"]
  extend-attention-kernel --> intel-amx["Intel AMX"]
  decode-attention-kernel --> intel-amx
  mla-optimization-layer --> intel-amx
  moe-kernel --> intel-amx
  moe-kernel --> avx512-kernels["AVX512 kernels"]
  fp8-gemm-path --> avx512-kernels

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`SGLang CPU backend`	Native CPU serving layer for large language models on Intel Xeon processors.	`Extend attention kernel`, `Decode attention kernel`, `MLA optimization layer`, `MoE kernel`, `FP8 GEMM path`, `Multi-NUMA tensor parallel runtime`
`Extend attention kernel`	Handles prefill attention over prefix and newly extended prompt tokens.	`SGLang CPU backend`, `Intel AMX`
`Decode attention kernel`	Splits KV sequences during generation to recover CPU parallelism when query length is one.	`SGLang CPU backend`, `Intel AMX`
`MLA optimization layer`	Packs shared key/value storage and folds heads for DeepSeek-style Multi-head Latent Attention.	`SGLang CPU backend`, `Intel AMX`
`MoE kernel`	Sorts activations by expert and fuses MoE projection, SiLU, and quantization work.	`SGLang CPU backend`, `Intel AMX`, `AVX512 kernels`
`FP8 GEMM path`	Emulates FP8 execution by unpacking weights to BF16 and caching converted blocks.	`SGLang CPU backend`, `AVX512 kernels`
`Multi-NUMA tensor parallel runtime`	Maps tensor parallel ranks onto CPU NUMA domains and uses shared-memory collectives.	`SGLang CPU backend`
`Intel AMX`	Matrix acceleration instruction set used by attention, MLA, and quantized GEMM kernels.	`Extend attention kernel`, `Decode attention kernel`, `MLA optimization layer`, `MoE kernel`
`AVX512 kernels`	Vectorized CPU path for pointwise operations, packing, and quantization support.	`MoE kernel`, `FP8 GEMM path`

Data flow

A model request enters SGLang's CPU backend. Prefill uses extend attention over prefix and extend query regions, decode attention splits KV sequences for parallelism, MLA kernels reuse shared key/value storage, MoE kernels sort and block expert activations, quantized GEMMs run through AMX/AVX512 paths, and tensor-parallel ranks are pinned across NUMA domains before SGLang returns generated tokens.

“This section focuses on 4 performance hotspots: Extend Attention and Decode Attention which are backends for RadixAttention of SGLang; MoE which contributes to the majority of weights in DeepSeek R1; and FP8 GEMM”article, stated

“To remove redundant computation, SGLang divides query sequence into two parts: * prefix – historical sequence in which attention is a rectangle; * extend – newly added prompt in which attention is a lower triangle.”article, stated

Why this architecture

The architecture attacks the specific CPU bottlenecks that make large MoE serving hard: memory footprint, MoE bandwidth, decode parallelism, FP8 support, and NUMA locality. It keeps SGLang as the serving framework while replacing GPU assumptions with CPU kernels and placement choices.

“Intel PyTorch Team contributed to CPU backend for SGLang for the past few months and we proposed a high-performance CPU only solution using 6th generation of Intel® Xeon ® Scalable Processor with only fractional cost.”article, stated

“To reduce remote memory access to minimal level, we mapped the Tensor Parallel (TP) for multi-GPU to multi-numa on a CPU server.”article, stated

Models used: DeepSeek-R1-671B · Qwen3-235B-A22B · DeepSeek-R1-Distill-Llama-70B · Llama-3.2-3B-Instruct

Scale

The target model class includes DeepSeek-R1-671B, whose deployment usually requires 8 to 16 high-end accelerators.
“The significant memory requirements will normally require 8x or even 16x high-end AI accelerators to deploy.”article, stated
The benchmark platform was a dual-socket Xeon 6980P server with 128 cores per socket.
“Our test platform is a state of art dual socket Intel® Xeon® 6980P CPU server, 128 cores each socket.”article, stated
INT8 MoE kernels reached 85% memory-bandwidth efficiency and 1.45 TB/s effective memory bandwidth.
“With these optimizations combined, we achieve 85% memory bandwidth efficiency for INT8 MoE, or 1.45TB/s effective memory bandwidth”article, stated
SGLang CPU backend achieved 6-14x TTFT speedup and 2-4x TPOT speedup against llama.cpp.
“TTFT achieved a 6-14x performance speedup. MoE models have larger improvements since experts are computed sequentially in llama.cpp and we parallel among experts by realigning expert indices.”article, stated

Alternatives considered (roads not taken)

Deploy DeepSeek R1 only on high-end accelerators

Rejected because: The article frames the accelerator requirement as the cost barrier the CPU-only solution is meant to reduce.

“However, its massive model size and unique architecture have posed new challenges on deployment. The significant memory requirements will normally require 8x or even 16x high-end AI accelerators to deploy.”article, stated

Use vanilla sequential MoE execution

Rejected because: Sequential expert loops waste CPU bandwidth and parallelism for MoE layers.

“A naïve implementation of MoE with torch would involve looping through experts sequentially, and gather (mask) activations for each of the expert before linear projection.”article, stated

Wait for native FP8 CPU support

Rejected because: Existing x86 platforms lacked FP8 support, so the team emulated FP8 to preserve the original user experience.

“DeepSeek R1 employs FP8 hybrid training, which is a great challenge for CPU devices, due to an obvious reason that existing x86 devices don’t have native support for FP8.”article, stated

Patterns

`latency-optimization` (performance)

The CPU backend targets TTFT and TPOT with specialized attention, MoE, FP8, and NUMA kernels.

“Achieve 6-14x speedup for TTFT and 2-4x for TPOT v.s. llama.cpp.”article, stated

`cost-aware-routing` (cost)

The deployment path makes large MoE serving possible on CPU hardware instead of defaulting to expensive accelerators.

“we proposed a high-performance CPU only solution using 6th generation of Intel® Xeon ® Scalable Processor with only fractional cost.”article, stated

`batch-offline-processing` (infrastructure)

The kernel work restructures tensor blocks, KV splits, expert chunks, and cached weight blocks so CPU inference can process large model work efficiently.

“we visit the weight blocks in a zigzag pattern, and cache the unpacked BF16 blocks in L2, make sure that the slow data type conversion for each block only happens once.”article, stated

Tradeoffs

Serve huge MoE models on CPU

Chose Xeon 6 CPU backend with AMX and quantized kernels over requiring 8x or 16x high-end accelerators.

Rationale: The CPU route lowers hardware cost while still meeting useful inference performance.

“we explain the technical details of achieving high efficiency for deploying DeepSeek on a single node with Xeon® 6 CPU.”article, stated

Emulate FP8 rather than omit it

Chose weight-only FP8 conversion and cache blocking over dropping FP8 model support on CPU.

Rationale: FP8 was needed for original model behavior, but the conversion path had to be optimized around CPU limitations.

“Notably both approaches are equally slow, take 60 to 70 cycles to accomplish, unacceptable for any performance critical scenario.”article, stated

Gotchas (what bit them)

Decode attention loses obvious parallelism when generating one token at a time.

Root cause: Single-request decoding collapses the query shape to one batch, one query block, and limited head-level parallelism.

Resolution: Chunk the KV sequence into multiple splits with Flash Decoding and reduce split outputs.

“Decoding faces more pressure on parallelization compared with prefilling, due to fact that query sequence length reduced to one.”article, stated

FP8 support on CPU is a performance and correctness compatibility problem.

Root cause: The x86 platform does not natively support FP8, but DeepSeek R1 was trained with FP8 behavior.

Resolution: Use weight-only FP8, vectorized conversion shortcuts, and cache-aware unpacking to approximate GPU FP8 behavior.

“providing FP8 support is essential since it represents the original user experience.”article, stated

NUMA placement can erase CPU serving gains.

Root cause: Remote memory access is slower when tensor-parallel work crosses CPU memory domains badly.

Resolution: Map tensor parallelism to multi-NUMA placement and use shared-memory communication primitives.

“Under NUMA, a processor can access its own local memory faster than remote memory”article, stated

Evaluation

The team benchmarked SGLang CPU backend against llama.cpp across four models, INT8/FP8/BF16 modes, socket settings, TTFT, TPOT, and kernel efficiency metrics.

Metrics: TTFT · TPOT · Memory bandwidth efficiency · Effective memory bandwidth · Kernel speedup · FP8 accuracy parity on GSM8K and MMLU

“We take another popular LLM tool llama.cpp as the performance baseline to compare against SGLang CPU backend. We evaluated 4 models range from 3B to 671B”article, stated

Outcomes

SGLang added native Intel Xeon CPU backend support.
The backend supported BF16, INT8, and FP8 for dense and sparse FFNs.
TTFT improved 6-14x and TPOT improved 2-4x versus llama.cpp.
INT8 MoE kernels reached 85% memory bandwidth efficiency.
Emulated FP8 reached approximately 80-90% of INT8 implementation performance.

Benchmarks included models from 3B to 671B parameters on a dual-socket Xeon 6980P system.

“Support BF16, INT8 and FP8 for both Dense FFNs and Sparse FFNs (MoE).”article, stated

“FP8 implementation achieves approximately 80% to 90% of INT8 implementation.”article, stated