High-Performance GPU Memory Transfer Optimization for Large Language Models

perplexity · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM infrastructure GPU networking

Perplexity needed to transfer non-contiguous GPU memory regions between machines at near-maximum AWS p5 network bandwidth while supporting asynchronous workloads and dynamic Kubernetes node changes.

Constraints:

• AWS p5 instances expose 3200 Gbps of network bandwidth through 32 network cards
• the workload needs high-bandwidth transfer between remote GPUs of non-contiguous memory chunks
• nodes must be dynamically added or removed from Kubernetes deployments without disrupting ongoing operations
• peer-to-peer communication patterns are required
• NCCL's static world requires cluster restarts when participating nodes change
• NCCL's synchronous communication model adds complexity for the asynchronous workload
• maximum bandwidth requires RDMA, GPU topology awareness, NUMA-aware allocation, thread pinning, batching, queuing, and completion polling

Architecture

Perplexity built a custom high-performance networking solution on AWS p5 instances using Elastic Fabric Adapter, RDMA, GPUDirect, and libfabric. Control messages use two-sided RDMA SEND/RECV to exchange memory-region metadata such as offsets and sizes, while the actual data plane uses one-sided RDMA WRITE operations where each write handles one contiguous GPU memory chunk. The implementation evolved from unidirectional SEND/RECV to bidirectional communication, direct GPU-to-GPU GPUDirect RDMA writes, multiple concurrent transfers, application-level operation queuing, and finally 32-network-card scaling with warmup, per-GPU network threads, CPU core pinning, state sharding, batching, lazy operation posting, and NUMA-aware libfabric resource allocation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Kubernetes deployment"] --> B["Transfer application"]
  B --> C["Application operation queue"]
  C --> D["Completion polling loop"]
  D --> E["libfabric layer"]
  B --> E
  E --> F["RDMA control plane"]
  F --> G["Memory registration"]
  E --> H["RDMA data plane"]
  G --> I["Source H100 GPU"]
  H --> J["EFA network cards"]
  I --> J
  J --> K["Destination H100 GPU"]
  B --> L["Topology-aware threading"]
  L --> J

Components

Data flow

The application registers GPU memory so CPU, GPUs, and network cards share the same virtual address understanding. Control-plane operations such as connection setup, memory registration, and SEND/RECV metadata messages go through the kernel where required for security. Data-plane transfer bypasses the kernel: a network card reads directly from source GPU memory, sends the bytes over EFA/SRD to the remote network card, and the remote card writes directly into destination GPU memory. The application maintains queues of pending operations, polls hardware completion queues, batches and lazily posts work to network cards, and pins network threads and libfabric resources to the correct NUMA locality.

Why this architecture

Perplexity chose a custom RDMA/libfabric implementation because its workload needed asynchronous peer-to-peer transfers, dynamic Kubernetes membership, and direct control over non-contiguous GPU memory movement. Traditional TCP/IP copies data through CPU and kernel buffers, while RDMA/EFA/GPUDirect lets network cards move data directly between GPU memories. NCCL is strong for collective communication, but its static world and synchronous model did not fit this dynamic asynchronous transfer problem.

Models used: large language models · deep learning workloads

Scale

• The target AWS p5 platform offers 3200 Gbps of network bandwidth through 32 network cards.
  “AWS p5 instances, offers an impressive 3200 Gbps of network bandwidth through 32 network cards.”article, stated
• The final implementation achieved 3,108 Gbps, or 97.1% of theoretical bandwidth.
  “Through these optimizations, we achieved a final performance of 3,108 Gbps - 97.1% of the theoretical maximum bandwidth.”article, stated
• A single network card reached 97.4% bandwidth utilization before scaling to 32 cards.
  “Achieved 97.4% bandwidth utilization on a single network card”article, stated
• Each p5 instance contains two CPU sockets, two NUMA nodes, and four PCIe switches per NUMA node.
  “each instance contains two CPU sockets forming two NUMA nodes, with each NUMA node connecting to four PCIe switches”article, stated
• Under each PCIe switch there are four 100 Gbps EFA network cards, one H100 GPU, and one NVMe SSD.
  “Four 100 Gbps EFA network cards”article, stated

  “One NVIDIA H100 GPU”article, stated
• The command-line demo reports 3108.283 Gbps transfer speed.
  “achieving a transmission speed of 3108.283 Gbps - demonstrating near-theoretical bandwidth utilization of the network infrastructure”article, stated

Alternatives considered (roads not taken)

Use NVIDIA NCCL

Rejected because: NCCL is excellent for collective distributed deep-learning communication, but its static world and synchronous model conflicted with Perplexity's dynamic, asynchronous workload.

Use traditional TCP/IP networking

Rejected because: TCP/IP requires multiple copies through user space, kernel space, and network buffers, consuming CPU cycles and memory bandwidth.

Let the kernel own network buffers and completion notifications

Rejected because: Kernel-managed buffers and epoll-style notifications add copies and system-call overhead that high-performance RDMA avoids.

Submit operations directly to network cards

Rejected because: Direct submission is less robust under congestion; an application-level queue simplifies the programming model and lets pending work be posted lazily after completions.

Patterns

latency-optimization (infrastructure)

Perplexity eliminated CPU/kernel data copies with RDMA, used GPUDirect GPU-to-GPU writes, polled completion queues in user space, pinned threads, and made resource allocation NUMA-aware.

fallback-degradation (robustness)

Application-level operation queuing and lazy posting provide robustness against congestion while maintaining efficient network-card utilization.

cost-aware-routing (resource utilization)

The system explicitly targets near-theoretical utilization of the expensive p5/EFA/H100 network topology by spreading work across all 32 network cards and minimizing wasted CPU/memory copies.

Tradeoffs

Build a custom solution instead of using NCCL

Chose custom libfabric/EFA/RDMA transfer layer over NVIDIA NCCL collective communication.

Rationale: Perplexity wanted asynchronous peer-to-peer transfer, dynamic Kubernetes membership, and direct control over memory-transfer patterns, which NCCL's static world and synchronous model did not fit.

Use RDMA/EFA rather than TCP/IP

Chose direct GPU-memory-to-network-card transfer over kernel-mediated TCP/IP socket copies.

Rationale: TCP/IP incurs repeated copies and context switches, while RDMA moves data directly between GPU memory and network cards.

Separate control plane from data plane

Chose kernel-mediated setup and metadata, kernel-bypassing data transfer over sending all operations through the same socket-style path.

Rationale: Security-sensitive setup still goes through the kernel, while the high-volume data path bypasses it for maximum performance.

Queue and lazily post operations

Chose application-level queue with completion-driven submission over immediate direct submission to network cards.

Rationale: Queuing improves robustness against congestion and lets the application submit pending operations when the completion queue indicates capacity is available.

Gotchas (what bit them)

NCCL's static world conflicts with dynamic Kubernetes membership.

Root cause: NCCL requires a fixed participating set for collective communication, so changing nodes requires restarting the whole cluster.

Resolution: Perplexity built a custom libfabric/RDMA transfer layer that supports dynamic add/remove behavior and peer-to-peer transfer patterns.

Zero-copy networking shifts buffer responsibility to the application.

Root cause: RDMA bypasses kernel-managed sockets, so the application must manage buffer ownership, memory registration, receive posting, and completion polling.

Resolution: The implementation registers memory, pre-posts receives, transfers ownership to network cards during operations, and polls completion queues directly.

Hardware topology can dominate achievable bandwidth.

Root cause: p5 topology has NUMA sockets, PCIe switches, EFAs, GPUs, and NVMe devices whose pairing affects whether transfers stay local or congest shared buses.

Resolution: Use proper hardware pairing, per-GPU threads, CPU core pinning, state sharding, and NUMA-aware libfabric resource allocation.

Non-contiguous memory complicates direct RDMA transfer.

Root cause: One RDMA WRITE handles one contiguous memory chunk, while the workload needs to transfer non-contiguous GPU memory regions.

Resolution: The custom solution sends control metadata about offsets and sizes and decomposes transfers into RDMA WRITE operations for contiguous chunks.

Evaluation

Perplexity evaluated the custom RDMA implementation through progressive performance milestones: basic SEND/RECV, bidirectional communication, GPUDirect RDMA WRITE, concurrent transfers, operation queuing, single-card utilization, and full 32-card scaling. Final results were reported as Gbps and percentage of theoretical bandwidth, including a command-line demo.

Metrics: 97.4% bandwidth utilization on a single network card · 3,108 Gbps final performance · 97.1% of theoretical maximum bandwidth · 3108.283 Gbps command-line demo transmission speed · 32 network cards used for scaling

Outcomes

• The custom networking solution achieved 3,108 Gbps, or 97.1% of theoretical maximum bandwidth.
• A command-line demo transferred non-contiguous GPU memory chunks at 3108.283 Gbps.
• The implementation reached 97.4% bandwidth utilization on a single network card before scaling to 32 cards.
• The technical deep dive and implementation details were made available in an open-source repository.

The solution targets AWS p5 infrastructure with 32 EFA network cards and 3200 Gbps theoretical bandwidth, optimizing peer-to-peer transfer of non-contiguous GPU memory for large-scale deep-learning and LLM workloads.