Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To overcome the semantic sparsity and noise of web-scale video captions, this work builds a multi-view captioning pipeline over 10 million videos that synthesizes reasoning-refined summary, detailed, and semantic-positive text supervisions. The authors couple this enriched corpus with a granularity-aware text architecture that employs distinct [CLS] tokens to disentangle high-level summaries from fine-grained spatiotemporal features. Pretraining on these multi-view captions yields consistent gains across standard, fine-grained, and detailed text-to-video retrieval benchmarks while operating on a significantly smaller data budget than competing methods.
Video foundation models don't need larger web scrapes to master fine-grained spatiotemporal understanding—they need multi-granularity synthetic captions paired with view-specific [CLS] embeddings.
Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/