Search papers, labs, and topics across Lattice.
The LAION-BVD dataset comprises 80 million videos totaling 10 million hours, collected from 1.3 billion URLs, aimed at enhancing multimodal learning across video, audio, and image modalities. By employing content-aware scene detection, the authors generate synthetic captions for video and audio clips, leading to models that outperform standard benchmarks as training scales increase. Additionally, the dataset includes scene-changing frames that provide a unique visual distribution, resulting in improved image-text retrieval performance, thus significantly advancing open access to multimodal video resources.
Models trained on LAION-BVD achieve state-of-the-art performance in multimodal tasks, showcasing the dataset's potential to redefine video understanding.
We present LAION-BVD, a large-scale open video dataset for multimodal learning, which contains 1.3B platform-specific video URLs collected from CommonCrawl. From these, we download 80M videos with a total duration of 10 million hours. The dataset is designed for multimodal pre-training across the video, audio, and image modalities. Using content-aware scene detection, we extract clips for which we synthetically generate video and audio captions. Models trained on these data achieve competitive performance on standard video-text and audio-text benchmarks, with consistent improvements as training or model scale increases. Additionally, we explore video frames as an alternative source of image-text data by extracting scene-changing frames. These frames exhibit a visual distribution distinct from standard web image corpora, and models trained on this dataset achieve strong image-text retrieval performance. We release LAION-BVD to the research community. It significantly expands open access to multimodal videos at an unprecedented scale.
