Search papers, labs, and topics across Lattice.
This paper introduces PILL (Probing-based InfiLling with preset-Length-free decoding), a novel method for adaptive-length infilling in diffusion language models (DLMs) that eliminates the need for a preset initial length and significantly reduces inference time. By addressing the limitations of prior methods, which are sensitive to initial length and computationally inefficient, PILL achieves superior performance across multiple benchmarks, improving pass rates and BLEU scores while being 1.82 times faster than existing approaches. The results demonstrate that PILL effectively enhances the flexibility and efficiency of DLMs in infilling tasks, making it a valuable advancement in the field.
PILL achieves up to 6.0 BLEU-2 improvement on text infilling while running 1.82x faster than the best existing method, revolutionizing efficiency in diffusion language models.
Diffusion language models (DLMs) have emerged as a promising alternative to the auto-regressive paradigm. With bidirectional attention and any-order generation, DLMs naturally fit infilling tasks, which require generating a middle span conditioned on both the prefix and the suffix. However, infilling is sensitive to the length of the span, while DLMs require the length to be fixed before generation. Although prior studies extend DLMs to dynamic lengths, they still suffer from two limitations. (i) Sensitivity to initial length. These methods require a preset length to initialize the search and are highly sensitive to this initial length, often yielding suboptimal results. (ii) Inference inefficiency. They either insert length-changing operations during generation or repeatedly search for an appropriate length using multi-step denoising confidence, both of which introduce substantial extra forward passes and computational cost. Therefore, we propose PILL (Probing-based InfiLling with preset-Length-free decoding), an efficient infilling method for DLMs that requires no preset initial length and adds far fewer extra forward passes than baselines, substantially reducing inference time. Experiments show that, across five DLMs spanning different families, architectures, and training recipes on eight infilling benchmarks, PILL improves over the strongest baseline by +4.8 average pass rate on code and +6.0 BLEU-2 on text, while running 1.82x faster than that baseline. The code is available at https://github.com/Hsu1023/PILL.
