The code for PTXBench is available here.

Why not directly generate CUDA-PTX?

PTX is the lowest-level GPU interface that CUDA programmers can explicitly control, so generating CUDA with inline, architecture-specific PTX offers the shortest path from a new hardware feature to a working kernel. This approach has traditionally looked unattractive: PTX is difficult to program and validate, while abstractions such as Triton and CuTeDSL provide productivity and portability. Yet those abstractions must continually absorb new instructions, layouts, and synchronization mechanisms through compiler engineering. As GPU architectures evolve faster and LLMs become better at code generation and iterative repair, directly generating CUDA-PTX becomes appealing as a way to use new hardware capabilities before the higher-level software stack fully catches up.

What does PTXBench ask?

PTXBench asks how well current LLMs can reason about architecture-specific PTX on H100 and B200 GPUs, not merely whether they can emit a fast CUDA kernel. A model receives an architecture-specific knowledge pack, writes CUDA-PTX, and revises it over multiple turns using compilation, sanitization, correctness, and performance feedback. The evaluation framework separately checks whether the kernel is functionally correct, whether the requested instruction family actually executes at runtime, and whether the kernel is competitive with frontier libraries. This separation also points to the techniques that will matter next: execution-grounded repair, targeted post-training, runtime instruction verification, and much stronger testing infrastructure.

Takeaway 1: GEMM is close; attention is not

Frontier LLMs are beginning to make architecture-specific PTX work, but capability falls sharply as the workload becomes more complex. GEMM is closest to being solved: Claude Opus 4.8 reaches 1.012x cuBLAS performance on Blackwell, while Gemini 3.1 Pro reaches 0.892x. Attention remains substantially harder, especially on Blackwell, and backward attention is harder still. PTXBench measures performance against frontier libraries: cuBLAS 13.1 for GEMM, cuDNN 9.20.0 for the primary attention workloads, and FlashInfer 0.6.14 for GQA. Speedup is the reference-library latency divided by the generated kernel’s latency, so 1.0x means matching the corresponding performance baseline and values above 1.0x mean surpassing it.

Figure 1 uses two different kinds of ratios. The x-axis (p) is a speedup threshold relative to the library baseline; the y-axis, Fast_p, is the metric proposed by KernelBench: the percentage of evaluated turns that produced a correct kernel with speedup greater than (p). For example, Fast_0.8 = 40% means that 40% of turns produced a correct kernel exceeding 0.8x baseline performance. The “Target inst.” rows additionally require the kernel to execute a selected target instruction at runtime.

Figure 1 Correct kernels meeting each speedup threshold (Fast_p) on Blackwell (top two panels) and Hopper (bottom two panels), before and after requiring selected target instructions to execute at runtime. GEMM is much further along than attention, particularly backward attention.

Takeaway 2: A small, well-constructed SFT dataset can pay off

Specializing a model for CUDA-PTX may not require an enormous corpus. PTXBench adapts Qwen3.6-27B with Fixit examples built from the base model’s failed kernels, execution feedback, teacher repairs, and synthesized repair rationales. The smallest balanced recipe that solves all five primary evaluation problems contains only 158 records, whereas the base model produces no correct kernel on any Hopper workload in the main model comparison. The improvement is meaningful but not universal: the adapted model transfers to GEMM, all four head-dimension-64 attention tasks, both head-dimension-96 forward tasks, and—at a much lower success rate—GQA, yet still fails on head-dimension-96 backward attention. Coverage, balance, and teacher quality therefore matter at least as much as raw dataset size.

Figure 2 Qwen3.6-27B-PTX-SFT generalizes beyond its four training tasks, but transfer remains uneven across head dimensions and attention variants.

Cross-language transfer is more mixed (Figure 3). On the same five Hopper workloads, using the same checkpoint to generate Triton lowers turn-level correctness relative to the base model on every workload, yet raises the best correct speedup on the causal variants from 0.238x to 0.632x for MHA-Fwd-Causal and from 0.037x to 0.331x for MHA-Bwd-Causal. The SFT recipe can therefore improve peak performance substantially even while making correct Triton kernels less likely.

Figure 3 PTX SFT reduces turn-level correctness under Triton transfer, but substantially improves the best correct speedup on both causal attention workloads.

Takeaway 3: Abstractions still matter, but cracks are appearing

Higher-level abstractions still provide a major advantage, especially on newer hardware, but that advantage is no longer universal. Under the same eight-turn refinement protocol with execution feedback, Triton reaches a higher best correct speedup than CUDA-PTX in 18 of 20 model–architecture–workload comparisons across Gemini 3.1 Pro and GPT-5.6 Sol, including all 10 Blackwell comparisons. The two exceptions are GPT-5.6 Sol on H100 causal MHA forward, where CUDA-PTX reaches 0.865x versus Triton’s 0.852x, and H100 causal MHA backward, where CUDA-PTX reaches 0.746x versus Triton’s 0.734x. The largest gaps appear in Blackwell attention: on B200 causal MHA backward, Gemini produces no correct CUDA-PTX kernel but reaches 0.437x with Triton, while GPT-5.6 Sol improves from 0.339x to 0.494x. Triton’s compiler-encoded optimizations and configuration tuning remain especially valuable on Blackwell, while the two H100 exceptions show that recent LLMs can already make direct CUDA-PTX outperform Triton in selected settings.

Figure 4 Fast_p distributions for Gemini 3.1 Pro and GPT-5.6 Sol generating Triton or CUDA-PTX kernels with the same eight-turn refinement protocol on H100 (top) and B200 (bottom). Solid lines denote CUDA-PTX and dashed lines denote Triton; dotted lines mark the best correct speedup.

Takeaway 4: Testing will become the bottleneck

Finding kernels that are both correct and safe requires repeatedly running expensive performance measurements. PTXBench reduces this cost by leveraging the temporary locality of kernel evaluation requests for parallel agent loops. Specifically, PTXBench runs a separate host process that caches workload state such as input tensors, reference outputs, and reference latencies in GPU memory. Reusing that state improves kernel evaluation throughput by 2.24x, as Figure 5 shows. Even with reuse, however, the cached path still takes 2.72x as long as kernel execution alone.

Figure 5 Reusing stable workload state makes evaluation 2.24x faster, but the cached path still takes 2.72x as long as kernel execution alone.

What’s Next?

In the future, two priorities follow. First, optimize the checkers. NVIDIA Compute Sanitizer’s racecheck, for example, can take more than 1000x as long as native execution, while LLM serving has benefited from far greater investment. Faster incremental checks, cache-aware sanitization, and better overlap between generation and profiling would let agents learn from more execution feedback in the same time. Second, robustify the checkers. Output comparison alone cannot catch memory-safety bugs, races, asynchronous-lifetime violations, or every runtime failure.¹ Future checkers must make these properties, runtime error tracing, and an explicit unknown state first-class signals. As kernel agents explore increasingly adversarial corners of CUDA semantics, checker speed will determine how quickly they improve, and checker robustness will determine whether their apparent wins are real.

Acknowledgments

PTXBench is a collaborative effort by Genghan Zhang, Yixin Dong, Chengze Fan, Zhichen Zeng, Yueming Yuan, Shaowei Zhu, and Kunle Olukotun. We are grateful to members of RadixArk, the SGLang community, and the Stanford Pervasive Parallelism Lab for their technical support, insightful discussions, and help. The project received generous support from the Gemini Academic Program and the Tinker Research Grant.