Search papers, labs, and topics across Lattice.
This paper addresses the challenge of applying reinforcement learning to Masked Diffusion Language Models (MDLMs) by introducing a two-stage action Markov Decision Process (MDP) that separately optimizes token placement and position remasking. By decomposing the policy gradient into distinct token and masking components, the authors achieve significant improvements in reasoning tasks, achieving state-of-the-art scores of 87.1% on the GSM8K benchmark and 53.4% on the MBPP coding benchmark. This work highlights the importance of considering both token predictions and the order of unmasking in enhancing the performance of MDLMs.
Optimizing both token placement and remasking decisions in MDLMs leads to groundbreaking improvements in reasoning and coding tasks.
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.
