Search papers, labs, and topics across Lattice.
VidPrism is introduced as a novel heterogeneous temporal Mixture-of-Experts (MoE) framework to improve image-to-video transfer learning by addressing expert homogenization in VLMs. It achieves this by deploying functionally specialized experts for spatial understanding and temporal modeling, fed by a content-aware, multi-rate sampling module that generates semantically rich and motion-focused representations. Experiments on video recognition benchmarks demonstrate state-of-the-art performance and effective expert specialization.
VLMs can achieve state-of-the-art video recognition by splitting temporal modeling experts into specialized roles for spatial understanding and motion processing.
With the rapid development of pre-training technologies, adapting large-scale Vision-Language Models (VLMs) for video understanding \emph{\ie} image-to-video transfer learning has become a dominant paradigm. To achieve superior performance, it raises as an effective strategy among recent advances to employ Mixture-of-Experts (MoE) to enhance VLMs' temporal modeling capabilities. However, conventional MoE designs suffer from expert homogenization, where all experts act as identical generalists, inefficiently learning spatio-temporal features from undifferentiated video streams. To overcome this problem, we propose VidPrism, a novel heterogeneous temporal Mixture-of-Experts framework. VidPrism pioneers a division of labor by deploying functionally specialized experts, each assuming a role ranging from spatial understanding to temporal modeling. To feed these specialists appropriately, we introduce a content-aware, multi-rate sampling module that dynamically generates streams ranging from semantically rich to motion-focused representations, providing specialized inputs for experts. Furthermore, a dynamic, bidirectional fusion mechanism enables synergistic information exchange between these pathways, leading to a comprehensive video representation. Extensive experiments on various video recognition benchmarks demonstrate that VidPrism achieves state-of-the-art performance and effectively fosters expert specialization. Our source code is available at \href{https://github.com/Lrrrr549/VidPrism.git}{https://github.com/Lrrrr549/VidPrism.git}.
