Search papers, labs, and topics across Lattice.
This paper introduces KATok, a transformer-based variational autoencoder that utilizes an adaptive token selector to optimize video representation by dynamically discarding uninformative tokens based on their content-richness. This approach addresses the limitations of conventional VAEs, which use fixed compression ratios that fail to adapt to the varying complexities of video data, ultimately enhancing both computational efficiency and visual quality. Empirical results demonstrate that KATok achieves state-of-the-art reconstruction and generation quality while significantly reducing spatio-temporal redundancy.
Adaptive tokenization in KATok leads to state-of-the-art video compression without sacrificing visual fidelity, challenging the fixed compression paradigm of traditional VAEs.
Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.
