Search papers, labs, and topics across Lattice.
This paper introduces token time continuous diffusion (TTCD), a novel diffusion language model that operates in continuous space and employs a unique mechanism of per-token times to enhance the generation process. By deterministically mapping Gaussian noise to a final token canvas without further sampling, TTCD mitigates inaccuracies associated with parallel sampling in discrete space, allowing for differentiated rates of token generation. The results show that TTCD significantly outperforms traditional discrete models in both unconditional and conditional generation tasks, particularly at high speedups, while also achieving notable improvements in Sudoku solving tasks.
TTCD achieves superior conditional generation performance and faster token processing by leveraging continuous space modeling and per-token timing, challenging the limitations of traditional discrete models.
In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.
