Search papers, labs, and topics across Lattice.
WorldVLM is introduced as a hybrid architecture combining Vision-Language Models (VLMs) for high-level scene understanding and World Models (WMs) for predicting environmental dynamics in autonomous driving. The VLM generates behavior commands to guide the WM, enabling context-aware actions. Experiments evaluate different conditioning strategies, providing insights into the challenges of this hybrid design.
WorldVLM bridges the gap between contextual reasoning and dynamic prediction by using VLMs to guide WMs, offering a more interpretable and context-aware approach to autonomous driving.
Autonomous driving systems depend on on models that can reason about high-level scene contexts and accurately predict the dynamics of their surrounding environment. Vision- Language Models (VLMs) have recently emerged as promising tools for decision-making and scene understanding, offering strong capabilities in contextual reasoning. However, their limited spatial comprehension constrains their effectiveness as end-to-end driving models. World Models (WM) internalize environmental dynamics to predict future scene evolution. Recently explored as ego-motion predictors and foundation models for autonomous driving, they represent a promising direction for addressing key challenges in the field, particularly enhancing generalization while maintaining dynamic prediction. To leverage the complementary strengths of context-based decision making and prediction, we propose WorldVLM: A hybrid architecture that unifies VLMs and WMs. In our design, the high-level VLM generates behavior commands to guide the driving WM, enabling interpretable and context-aware actions. We evaluate conditioning strategies and provide insights into the hybrid design challenges.
