Search papers, labs, and topics across Lattice.
This paper introduces GenFirst, a novel approach to latent generative modeling that prioritizes generation before reconstruction, addressing the challenges of latent collapse and generation-reconstruction conflicts in traditional two-stage pipelines. By leveraging insights on the importance of entropy in the Kullback-Leibler divergence objective and the asymmetric learning dynamics between reconstruction and generation, the authors successfully implement an end-to-end training method that maintains latent space stability. The results show that GenFirst achieves state-of-the-art performance on benchmarks like ImageNet-256 and demonstrates versatility across various generative tasks, including text-to-image generation.
Generation-first training eliminates latent collapse, achieving unprecedented stability and performance in latent generative modeling.
Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.
