Automated GPU Kernel Generation Using LLMs and Inference-Time Scaling

nvidia · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: GPU systems

NVIDIA wanted to automate optimized attention-kernel generation, a specialized low-level programming task that normally requires expert GPU engineering and iterative performance tuning.

Constraints:

• attention kernels must be numerically correct as well as efficient
• attention variants such as causal masks, relative positional embeddings, ALiBi, full masks, and softcap need different implementations
• LLMs can hallucinate code, mix language syntax, and choose poor GPU thread mappings
• the workflow is experimental and not yet consistently reliable for all problem varieties

Architecture

NVIDIA built a closed-loop inference-time-scaling workflow for GPU kernel generation. A manually crafted prompt describes the target attention kernel, DeepSeek-R1 generates the initial GPU code, and a verifier running on H100 hardware tests or analyzes the generated kernel. If the kernel fails or can improve, Hopper GPUs refine and format feedback into a new prompt for DeepSeek-R1. The loop runs for a predetermined budget, about 15 minutes in the article, so the model can explore multiple candidate kernels and converge on numerically correct optimized implementations rather than relying on a one-shot answer.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  manual-kernel-prompt[Manual kernel prompt] --> deepseek-r1-generator[DeepSeek-R1 generator]
  deepseek-r1-generator --> h100-verifier[H100 verifier]
  h100-verifier --> feedback-prompt-formatter[Feedback prompt formatter]
  feedback-prompt-formatter --> deepseek-r1-generator
  h100-verifier --> kernelbench-benchmark[KernelBench benchmark]
  deepseek-r1-nim-microservice[DeepSeek-R1 NIM microservice] --> deepseek-r1-generator

Components

Data flow

An engineer supplies a technical prompt describing an attention variant and implementation constraints. DeepSeek-R1 generates GPU kernel code. A verifier on an NVIDIA H100 checks the generated kernel and creates feedback prompts from the result. Those prompts are fed back into DeepSeek-R1 for additional attempts until the time budget ends, after which the workflow selects the improved generated kernel and evaluates it against KernelBench correctness and speedup measurements.

Why this architecture

NVIDIA uses inference-time scaling because one-shot code generation is not reliable enough for GPU kernels. Attention implementations require precise math, memory behavior, and thread mapping, while LLMs can hallucinate or mix syntax. The verifier loop adds a concrete execution-based signal and turns failures into fresh prompts, spending more inference compute when the alternative is expert human kernel engineering. KernelBench provides a domain-specific correctness target, and NIM is the path toward operationalizing the model serving piece.

Models used: DeepSeek-R1 · DeepSeek-R1 NIM microservice

Scale

• Attention complexity grows quadratically with input sequence length.
  “The computational complexity of the attention operation grows quadratically in relation to the input sequence length.”article, stated
• The loop ran for about 15 minutes to improve attention kernels.
  “The team found that by letting this process continue for 15 minutes resulted in an improved attention kernel. !”article, stated
• KernelBench results reached 100% numerical correctness for Level-1 problems and 96% for Level-2 problems.
  “This workflow produced numerically correct kernels for 100% of Level-1 problems and 96% of Level-2 problems, as tested by [Stanford’s KernelBench](https://scalingintelligence.”article, stated
• More than 10 minutes per Level-1 problem produced numerically correct code for most of the 100 benchmark problems.
  “Allocating more than 10 minutes per problem in the Level-1 category enables the workflow to produce numerical correct code for most of the 100 problems.”article, stated
• Reported speedups over the PyTorch API baseline ranged from 1.1x to 2.1x across attention kernel types.
  “The PyTorch API maintains a baseline of 1x for all kernels, while the NVIDIA Workflow with DeepSeek-R1 achieves speedups of 1.1x for Causal Mask and Document Mask, 1.5x for Relative Positional, 1.6x for Alibi Bias and Full Mask, and 2.1x for Softcap.”article, stated

Alternatives considered (roads not taken)

One-shot LLM kernel generation

Rejected because: DeepSeek-R1 and similar models can generate useful code but still struggle to produce optimized, correct kernels on the first try.

Manual expert kernel development only

Rejected because: Creating optimized attention kernels takes significant skill and time, even for experienced engineers, so automated generation can reduce the human kernel-writing burden.

Accept simple attention implementations

Rejected because: Naive implementations can cause out-of-memory errors or poor computational efficiency as sequence length grows.

Patterns

eval-driven-development (verification)

Generated kernels are checked by a verifier and benchmarked for numerical correctness rather than accepted from the model directly.

prompt-chaining (generation-workflow)

Verifier feedback becomes fresh prompts that guide subsequent DeepSeek-R1 attempts in a loop.

latency-optimization (performance)

The generated kernels target attention speedups and computational efficiency on GPU workloads.

Tradeoffs

Inference-time scaling for code generation

Chose spending additional inference compute and verifier iterations over single-pass model generation.

Rationale: The extra compute lets the workflow explore multiple outcomes and improve correctness for a complex technical task.

Execution-backed verification

Chose H100 verifier and KernelBench correctness checks over static review or human inspection alone.

Rationale: GPU kernels can fail through subtle numerical or performance issues, so hardware-backed verification supplies concrete feedback.

Domain-expert prompting remains in the loop

Chose manual prompts with detailed formulas and implementation constraints over fully autonomous natural-language task discovery.

Rationale: The sample prompt shows that useful kernel generation still depends on specific mathematical and implementation guidance.

Gotchas (what bit them)

LLM-generated GPU code can be syntactically or semantically wrong.

Root cause: Models may hallucinate code or mix syntax from different languages and frameworks.

Resolution: NVIDIA runs generated kernels through an H100 verifier and feeds failures back into further prompts.

Correct GPU thread mapping is hard for a model to infer directly.

Root cause: Optimal thread mapping is non-trivial and often requires refinement based on hardware execution behavior.

Resolution: The closed-loop verifier supplies iterative guidance rather than trusting the first generated mapping.

The approach is promising but not universally consistent yet.

Root cause: Success rates still leave unsolved cases, especially outside the easier benchmark categories and problem variants.

Resolution: NVIDIA frames the work as active research and says more work is needed for wider problem consistency.

Evaluation

NVIDIA evaluates generated kernels with an H100 verifier and Stanford KernelBench, using numerical correctness and speedup comparisons against PyTorch/Flex Attention baselines.

Metrics: 100% numerical correctness on Level-1 KernelBench problems · 96% numerical correctness on Level-2 KernelBench problems · 1.1x-2.1x speedup over PyTorch API baseline depending on kernel type · time budget over 10 minutes and about 15 minutes in the closed loop

Outcomes

• The workflow produced numerically correct kernels for all Level-1 and 96% of Level-2 KernelBench problems.
• Some generated kernels outperformed skilled-engineer kernels in the reported experiment.
• The work remains research-oriented; NVIDIA explicitly says broader consistency still needs improvement.

The reported evaluation spans KernelBench Level-1 and Level-2 problems, including 100 Level-1 problems and multiple attention-kernel variants.