Search papers, labs, and topics across Lattice.
This paper introduces MDLMPE, a novel positional encoding method tailored for masked diffusion language models (MDLMs) that accounts for the dynamic configurations of revealed and masked tokens. By representing token availability as a binary sequence and employing distance-aware Gaussian weighting, MDLMPE enhances the positional context, significantly improving performance across various tasks such as fine-tuning and zero-shot evaluation. Experimental results demonstrate that MDLMPE consistently outperforms traditional positional encoding methods, highlighting the importance of adapting positional representations to the unique characteristics of MDLMs.
MDLMPE reveals that adapting positional encodings to the dynamic token availability in masked diffusion models can lead to substantial performance gains across multiple tasks.
Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.
