Search papers, labs, and topics across Lattice.
PTXBench is a newly introduced benchmark designed to evaluate and adapt large language models (LLMs) for GPU kernel optimization using architecture-specific PTX. The study reveals that while some models can execute target instructions, their performance does not consistently match that of leading libraries, particularly struggling with complex attention backward workloads. Additionally, adapting the Qwen3.6-27B model through supervised fine-tuning shows improvements in specific tasks, but highlights challenges in generalization tied to data quality and coverage.
Despite executing target instructions, LLMs often fail to deliver competitive performance in GPU kernel optimization, especially on complex tasks.
We introduce PTXBench, a benchmark for evaluating and adapting large language models (LLMs) to use architecture-specific PTX for GPU kernel optimization. PTXBench measures functional correctness, whether selected target instructions execute at runtime, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs. Our evaluation shows that architecture-specific PTX capability remains uneven: success rates fall substantially on complex attention backward workloads, and executing the target instructions does not necessarily translate into competitive performance. No evaluated model consistently matches frontier libraries across the suite. We further adapt Qwen3.6-27B using supervised fine-tuning. Repair-conditioned training improves several tasks, but generalization remains uneven; data coverage, balance, and the quality of the reasoning teacher matter in addition to dataset size. PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures.
CMU ML
Stanford HAI