Search papers, labs, and topics across Lattice.
This paper introduces Vega, a unified framework that effectively integrates video understanding and generation by leveraging a shared vocabulary for text and visual representations. By employing a hybrid architecture that combines autoregressive prediction for keyframes with diffusion-based rendering for high-resolution video frames, Vega addresses the inherent challenges of balancing compact semantic representations with the need for detailed temporal coherence. Extensive experiments show that Vega outperforms existing methods on both video generation and understanding benchmarks, highlighting its versatility and effectiveness in multimodal modeling.
Vega bridges the gap between video understanding and generation, achieving state-of-the-art results on multiple benchmarks by harmonizing semantic representation with visual detail.
Recently, unified image generation and understanding have been extensively explored. However, extending such unified modeling paradigms to the video domain remains largely underexplored. A central challenge is that video understanding favors compact, discriminative semantic representations, whereas video generation requires dense signals that preserve visual details and temporal coherence. Videos naturally capture both spatial semantics and temporal dynamics, making them a more suitable modality for unified multimodal modeling compared to static images. In this paper, we propose Vega, a unified framework that bridges video understanding and generation. Vega leverages a shared vocabulary to jointly model text and visual representations and employs a hybrid architecture combining autoregressive (AR) prediction with diffusion-based rendering. Specifically, the AR model focuses on predicting semantically meaningful visual tokens for keyframes, providing a structured representation that guides the diffusion module in rendering dense, high-resolution video frames. Extensive experiments demonstrate that Vega achieves strong performance on video generation benchmarks such as VBench and video understanding benchmarks like VideoMME.
