Search papers, labs, and topics across Lattice.
This paper critiques the generative perplexity (Gen-PPL) metric used in continuous diffusion language models, revealing that low Gen-PPL scores are often misleading due to excessive repetition in generated text. By stripping away this repetition, the authors demonstrate that the Gen-PPL can increase significantly, indicating a lower quality than initially perceived. They introduce ACE (Attractor-Contrast-Escape), a method that effectively reduces repetition in the self-conditioning feedback loop while maintaining competitive quality and significantly lowering computational costs.
Low generative perplexity in diffusion models often masks excessive repetition, with a simple fix cutting repetition to human levels while being 1.5–5x cheaper.
Continuous diffusion language models such as ELF report record-low generative perplexity (Gen-PPL). We find a catch: these models repeat far more than human text, and Gen-PPL rewards rather than penalizes that repetition, so its low scores overstate quality. Strip the repetition and ELF-B's Gen-PPL rises from $19.5$ to $27.7$; the smallest model even posts the best Gen-PPL because it repeats most. We trace the repetition to its source: a contractive attractor along a \emph{single direction} in the self-conditioning feedback loop, the loop that feeds each step's clean estimate into the next. Because the failure is one-dimensional, a one-dimensional fix suffices, and we propose one. \textbf{ACE} (Attractor-Contrast-Escape) subtracts that single, label-free direction from the feedback at each step. Estimated once on the $105$M model, the direction cuts repetition to near the human level while keeping quality competitive, and transfers near-unchanged to the $342$M and $652$M models and across samplers; the same recipe recovers useful directions on other architectures. Since Gen-PPL itself rewards repetition, we instead measure the compute each fix needs to produce human-clean text, where ACE is $1.5$--$5\times$ cheaper.
