Search papers, labs, and topics across Lattice.
This paper introduces VoidPadding, a novel approach that separates the roles of padding and semantic termination in Masked Diffusion Language Models (MDLMs) by utilizing a dedicated padding token, [VOID], while reserving the [EOS] token solely for indicating the end of a sequence. This separation addresses the issue of [EOS] overflow during large-block decoding, which has hindered performance in instruction tuning. The results demonstrate that VoidPadding significantly enhances performance on mathematical reasoning and code generation tasks, achieving a +17.84 point improvement over the baseline model and reducing decoding NFE by 55.7%.
Separating padding from semantic termination boosts performance by nearly 18 points in MDLMs while slashing decoding inefficiencies by over half.
MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by \(+17.84\) points over the original model and \(+6.95\) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.
Tsinghua AI