Search papers, labs, and topics across Lattice.
This paper introduces a continuous-time Markov chain framework for Insertion Language Models (ILMs), deriving a diffusion-style denoising objective that enhances insertion-based generation methods. By framing the noising process as a continuous-time Markov chain, the authors unify previous ad-hoc formulations and demonstrate that their approach retains the advantages of ILMs while competing effectively with left-to-right and masked diffusion models in language modeling tasks. Empirical evaluations reveal that this framework not only preserves the benefits of insertion-based generation but also provides greater flexibility in sampling strategies.
Insertion Language Models can achieve competitive performance with left-to-right and masked diffusion models while offering enhanced sampling flexibility through a novel continuous-time Markov chain framework.
Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.
