Search papers, labs, and topics across Lattice.
The paper introduces Guided Denoiser Self-Distillation (GDSD), a novel reinforcement learning method for fine-tuning diffusion language models (dLLMs) that avoids biases introduced by evidence lower bound (ELBO) approximations of the policy likelihood. GDSD distills the dLLM's denoiser directly from an advantage-guided self-teacher using a normalization-free objective, effectively reducing RL to likelihood-free self-distillation. Experiments on planning, math, and coding benchmarks demonstrate that GDSD outperforms state-of-the-art ELBO-based methods, achieving up to 19.6% improvement in test accuracy and exhibiting more stable training reward dynamics.
Ditch the ELBO: bypassing biased likelihood approximations in RL fine-tuning of diffusion LMs unlocks more stable and effective policy optimization, yielding nearly 20% accuracy gains on challenging tasks.
Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of methods replaces the likelihood in standard RL with its evidence lower bound (ELBO), estimated from randomly masked sequences. Despite being well aligned with pre-training, these approaches introduce bias through training--inference mismatch by using the ELBO as a likelihood surrogate, which can degrade performance. In this work, we propose Guided Denoiser Self-Distillation (GDSD) to directly distill the denoiser of dLLMs from an advantage-guided self-teacher, derived from the closed-form optimum of reverse-KL regularized RL. GDSD matches the dLLM's denoiser logits to the teacher's via a normalization-free objective, which reduces RL to likelihood-free self-distillation and thus bypasses the TIM biases. Recent ELBO-based methods emerge as instances of applying different distillation divergences, but with diagnosable pathologies that GDSD avoids. On planning, math, and coding benchmarks with LLaDA-8B and Dream-7B, GDSD consistently outperforms prior state-of-the-art ELBO-based methods with a more stable training reward dynamics, achieving test-accuracy improvements of up to $+19.6\%$. These results suggest that direct denoiser self-distillation, without relying on an ELBO likelihood surrogate, can provide a more stable and effective RL procedure for dLLMs. Code is available at https://github.com/GaryBall/GDSD.
