Search papers, labs, and topics across Lattice.
Sekai2 is a novel multi-source real-world video dataset designed to enhance interactive world modeling by providing long-horizon generation and camera control through extensive video clips. It comprises 128,892 clips totaling 2,826 hours from diverse global sources, with a focus on sustained observation and detailed annotations that include camera trajectories and hierarchical motion dynamics. The dataset's unique feature of revisiting locations across time and viewpoints facilitates the learning of persistent scene representations and geometrically consistent world models, making it a vital resource for advancing video generation techniques.
Revisiting the same locations across time in Sekai2 enables the learning of persistent scene representations, a game-changer for interactive world modeling.
Video world models must capture how scenes evolve over time and across viewpoints. Training them for long-horizon generation and camera control therefore benefits from long videos paired with camera trajectories and temporally grounded semantics. Existing corpora rarely offer the three together: large-scale web video provides broad visual diversity but no trajectories or time-aligned text, while pose-annotated datasets are typically short-range or reconstruction-oriented. We introduce Sekai2, a multi-source real-world video dataset that carries the world-exploration footage of Sekai toward interactive world modeling. The release contains 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries or regions, and is deliberately weighted toward sustained observation: under a common 120-second decomposition, 43,594 segments reach the full two minutes and account for 51.4% of all footage. Every clip includes a released camera trajectory and hierarchical annotations disentangling subject motion, environment dynamics, static scene content, and camera behavior, resulting in 649,597 temporally grounded segments. Crucially, we further introduce 982 panoramic sequences captured along non-linear trajectories with loops and revisits. These revisits provide repeated observations of the same locations across time and viewpoints, offering essential supervision for learning persistent scene representations, long-term spatial memory, and geometrically consistent world models. Corpus-scale analyses demonstrate complete pose-and-caption coverage, broad geographic and semantic diversity, varied camera trajectories, and highly non-redundant temporal descriptions. Together, these properties make Sekai2 a scalable resource for long-horizon video generation, camera-controllable synthesis, and interactive world-model pre-training.
Tsinghua AI