Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TAME, a novel framework for Text-Video Retrieval that enhances traditional CLIP models by incorporating temporal modeling through a combination of sparse Mixture-of-Experts (MoE) layers and Frame-Temporal (FT) tokens. By allowing experts to specialize in frame-level visual patterns and aggregating global cross-frame information, TAME effectively captures long-range temporal dependencies while maintaining local detail. Experimental results demonstrate significant improvements over existing CLIP-based methods, achieving a 4.0 increase in R@1 on the MSR-VTT benchmark and consistent gains across multiple datasets.
TAME achieves a 4.0 increase in retrieval accuracy on MSR-VTT by effectively modeling temporal dynamics in video data, outpacing traditional CLIP-based approaches.
Text-Video Retrieval (TVR) retrieves videos that match a natural-language query, but extending image-text models such as CLIP to videos is fundamentally limited by the lack of temporal modeling. Videos exhibit frame-wise heterogeneity in appearance and motion, and compressing all frames into a single representation often obscures temporal structure and semantic transitions. To address this, we propose Temporal-Aware Mixture-of-Experts for Text-Video Retrieval (TAME), a CLIP-based framework that jointly models frame-level structure and temporal relations. First, we integrate sparse Mixture-of-Experts (MoE) layers into both CLIP encoders and apply frame-consistent routing on the vision branch so that experts specialize according to frame-level visual patterns while preserving the original vision-language alignment. Second, we introduce Frame-Temporal (FT) tokens that aggregate global cross-frame information and feed it back to each frame, enabling the visual encoder to capture long-range temporal dependencies without harming local details. Third, we design a Cross-Temporal Interaction and Aggregation (CTIA) module that refines frame-wise sentence-video similarities through staged temporal filtering and fusion. Experiments on standard TVR benchmarks show that TAME consistently improves over CLIP-based baselines. On MSR-VTT, it improves R@1 by 4.0 over CLIP4Clip, and also achieves consistent gains on DiDeMo, MSVD, LSMDC, and ActivityNet. The code is available at https://github.com/sejong-rcv/TAME.