Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VideoRAE, a representation autoencoder that transforms frozen representations from Video Foundation Models (VFMs) into compact, generation-friendly video latents. By leveraging multi-scale hierarchical features and a lightweight 1D self-attention projector, VideoRAE achieves strong semantic preservation during decoding without the need for KL regularization. Experimental results demonstrate that VideoRAE outperforms existing autoencoder baselines in terms of reconstruction quality and convergence speed, achieving state-of-the-art performance on UCF-101 with significant efficiency gains.
VideoRAE reveals that frozen Video Foundation Model representations can be effectively transformed into high-quality, generation-friendly video latents, achieving state-of-the-art results with faster convergence.
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.