Search papers, labs, and topics across Lattice.
The paper introduces DecQ, a novel framework for Representation Autoencoders (RAEs) that enhances both reconstruction quality and generative performance by incorporating lightweight detail-condensing queries. These queries extract fine-grained information from intermediate layers of frozen vision foundation models (VFMs) and are integrated into the decoder. Experiments show that DecQ significantly improves reconstruction PSNR and accelerates generative modeling convergence while maintaining high FID scores, effectively mitigating the trade-off between reconstruction and generation.
Freezing your vision foundation model doesn't have to mean sacrificing fine-grained detail: DecQ unlocks improved reconstruction and faster generative convergence with just 8 extra queries and minimal overhead.
Representation Autoencoders (RAEs) leverage frozen vision foundation models (VFMs) as tokenizer encoders, providing robust high-level representations that facilitate fast convergence and high-quality generation in latent diffusion models. However, freezing the VFM inherently constrains its spatial reconstruction capacity, limiting fine-grained generation and image editing; in contrast, incorporating reconstruction-oriented signals via fine-tuning disrupts the pretrained semantic space and degrades generative fidelity. To address this trade-off, we propose DecQ, a simple yet effective framework for RAEs. Specifically, DecQ introduces lightweight detail-condensing queries that extract fine-grained information from intermediate VFM features through condenser modules. These queries are incorporated into the decoder to support reconstruction and are jointly generated with patch tokens during generative modeling. By aggregating information from both shallow and deep layers, DecQ effectively mitigates the reconstruction--generation trade-off, improving both reconstruction quality and generative performance. Our experiments demonstrate that: (1) with only 8 additional queries and 3.9% extra computation, DecQ improves reconstruction over the frozen DINOv2-based RAE, increasing PSNR from 19.13 dB to 22.76 dB; and (2) for generative modeling, DecQ achieves 3.3times faster convergence than RAE, attaining an FID of 1.41 without guidance and 1.05 with guidance.
