Search papers, labs, and topics across Lattice.
Orca is introduced as a general world foundation model that learns a unified latent space from multimodal signals, utilizing Next-State-Prediction modeling to enhance understanding and interaction with the world. The model is trained through both unconscious learning from continuous video data and conscious learning from language-described events, leveraging a large-scale dataset of 125K hours of video and 160M event annotations. Evaluation across text generation, image prediction, and embodied action generation demonstrates that Orca's unified latent space significantly improves performance over specialized baselines, highlighting its potential as a versatile tool for AI applications.
Orca's unified world latent space enables superior performance in diverse tasks, outperforming specialized models with a single framework.
We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.
