Search papers, labs, and topics across Lattice.
The paper introduces Dream-Coder 7B, a discrete diffusion language model for code generation capable of any-order generation, adapting its decoding strategy based on the coding task. They convert a pretrained autoregressive model into a diffusion model using a continuous-time weighted cross-entropy objective and address padding issues with random truncation and a padding penalty during supervised fine-tuning. The model is further refined using reinforcement learning with verifiable rewards on a curated prompt set, achieving 21.4\% pass@1 on LiveCodeBench.
Forget left-to-right: Dream-Coder 7B's diffusion approach lets it generate code in *any* order, adapting its strategy to the task at hand.
We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.
