Search papers, labs, and topics across Lattice.
This paper investigates the quality breakdowns in compressed short-text generation by isolating failure modes in a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). Using a staged validation protocol, the authors reveal that the majority of quality loss occurs before latent generation, with codec reconstruction significantly increasing perplexity metrics. The findings highlight that improving codec fidelity is crucial for enhancing overall generation quality, rather than focusing solely on latent denoising techniques.
Codec fidelity, not latent generation, is the primary determinant of quality in compressed short-text generation, with reconstruction metrics showing dramatic improvements.
Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 $(\mathbf{+ 8 0. 4 \%})$ and $p 95$ from 25.10 to $98.91(+294.1 \%)$, showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and $p 95$ by $\mathbf{3 2. 9 \%, } \mathbf{3 0. 9 \%}$, and $\mathbf{3 6. 6 \%}$, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.
