Search papers, labs, and topics across Lattice.
This paper introduces Gen4U, a novel framework that leverages state-of-the-art video diffusion models to unify video generation and understanding without the need for fine-tuning. By analyzing intermediate activations through mutual-kNN alignment metrics, the authors reveal a structured latent space that captures both global semantics and fine-grained details across varying noise levels. The results demonstrate that these frozen models serve as effective video encoders, achieving competitive performance on diverse tasks such as video classification and captioning while maintaining their generative capabilities.
Frozen video diffusion models can effectively serve as competitive encoders for a wide range of tasks, merging generation and understanding seamlessly.
Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.
DeepMind