Search papers, labs, and topics across Lattice.
This paper introduces a unified synthetic data generation pipeline for multimodal video understanding, capable of producing diverse and richly annotated video data across tasks like object counting, VQA, and segmentation. To improve reasoning, they fine-tune models using a VQA-based strategy that encourages deeper visual grounding. Experiments show models trained on this synthetic data generalize well to real-world datasets, often surpassing models trained on real data, demonstrating the pipeline's effectiveness as a scalable alternative to real-world annotation.
Forget painstakingly annotating real-world videos: this unified synthetic data pipeline can generate multimodal video datasets that actually *improve* model performance on downstream tasks.
Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, and segmentation. However, collecting and annotating multimodal video data in real-world is costly, slow, and inherently limited in diversity and coverage. To address this challenge, we propose a unified synthetic data generation pipeline capable of automatically producing unlimited multimodal video data with rich and diverse supervision. Our framework supports multiple task formats within a single pipeline, enabling scalable and consistent data creation across tasks. To further enhance reasoning ability, we introduce a VQA-based fine-tuning strategy that trains models to answer structured questions about visual content rather than relying solely on captions or simple instructions. This formulation encourages deeper visual grounding and reasoning. We evaluate our approach in three challenging tasks: video object counting, video-based visual question answering, and video object segmentation. Experimental results demonstrate that models trained predominantly on synthetic data generalize effectively to real-world datasets, often outperforming traditionally trained counterparts. Our findings highlight the potential of unified synthetic data pipelines as a scalable alternative to expensive real-world annotation for multimodal video understanding.
