Search papers, labs, and topics across Lattice.
The paper introduces DICE, a series of diffusion large language models (dLLMs) specifically designed for CUDA kernel generation, addressing the challenges of specialized code generation and limited high-quality training data. To train DICE, the authors created CuKe, a supervised fine-tuning dataset, and developed a bi-phase curated reinforcement learning (BiC-RL) framework involving CUDA kernel infilling and end-to-end generation. Experiments on KernelBench demonstrate that DICE outperforms both autoregressive and diffusion LLMs, achieving state-of-the-art results in CUDA kernel generation across three parameter scales (1.7B, 4B, and 8B).
Diffusion LMs can now beat autoregressive models at CUDA kernel generation, thanks to a new dataset and reinforcement learning approach.
Diffusion large language models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) LLMs, owing to their capacity for parallel token generation. This paradigm is particularly well-suited for code generation, where holistic structural planning and non-sequential refinement are critical. Despite this potential, tailoring dLLMs for CUDA kernel generation remains challenging, obstructed not only by the high specialization but also by the severe lack of high-quality training data. To address these challenges, we construct CuKe, an augmented supervised fine-tuning dataset optimized for high-performance CUDA kernels. On top of it, we propose a bi-phase curated reinforcement learning (BiC-RL) framework consisting of a CUDA kernel infilling stage and an end-to-end CUDA kernel generation stage. Leveraging this training framework, we introduce DICE, a series of diffusion large language models designed for CUDA kernel generation, spanning three parameter scales, 1.7B, 4B, and 8B. Extensive experiments on KernelBench demonstrate that DICE significantly outperforms both autoregressive and diffusion LLMs of comparable scale, establishing a new state-of-the-art for CUDA kernel generation.
