Search papers, labs, and topics across Lattice.
This paper identifies a critical mismatch in diffusion language models (dLLMs) during pretraining, where both prompt and continuation tokens can be randomly corrupted, undermining effective prompt-conditioned generation. To address this, the authors propose a novel pretraining objective called Prefix-Conditioned Diffusion (PCD), which integrates autoregressive prefix supervision with suffix denoising, thereby aligning the training interface with evaluation conditions. Experimental results demonstrate that PCD significantly enhances performance on benchmark tasks, achieving a 4.2% relative improvement on LLaDA2-Mini and a 14.2% gain in comparisons with the Qwen model, indicating that better alignment of pretraining context can close the dLLM continuation gap.
Aligning pretraining with prompt-conditioned generation can recover significant performance in diffusion language models without altering inference methods.
Autoregressive language models align training and use: generation conditions on a clean prompt, and training predicts future tokens from clean left context. Diffusion language models offer parallel denoising, but native dLLM pretraining can randomly corrupt prompt and continuation tokens together, weakening the clean-prefix interface needed for prompt-conditioned generation. We identify this mismatch for prompt continuation and propose PCD (Prefix-Conditioned Diffusion), a pretraining objective that combines AR prefix supervision with no-shift suffix denoising. At the training-objective level, PCD changes the attention mask, corruption mask, and label construction in continued pretraining; it does not require an autoregressive decoder, verifier, or new inference mode. By supervising the clean-prefix side autoregressively and applying diffusion only to the unknown continuation, PCD makes the local training interface resemble how block-diffusion models are queried at evaluation time. We further separate intra-sample prefix conditioning from inter-sample objective mixing, allowing us to identify the local alignment signal separately from the optional batch-level mixing knob. Across LLaDA2-Mini and Qwen-1.7B backbones, PCD consistently improves over same-family native dLLM stable baselines, reaching a 4.2% relative gain on the main LLaDA2-Mini six-benchmark average (+2.56 points) and a 14.2% relative gain in the primary Qwen mechanism comparison (+4.86 points). These results suggest that aligning the pretraining context distribution with prompt-conditioned generation can recover a measurable part of the dLLM continuation gap without changing inference.
