Search papers, labs, and topics across Lattice.
This paper introduces VLM-CASE, a novel framework that equips autonomous vehicles with anticipatory capabilities to navigate adverse driving conditions by integrating a vision-language model (VLM) with a context-adaptive safety envelope (CASE). By fine-tuning the VLM using low-rank adaptation (LoRA), the system assesses road surface and visibility, allowing for proactive adjustments in speed and steering while adhering to a formal safety model. In closed-loop CARLA simulations, VLM-CASE-MPC demonstrated superior performance compared to conventional model predictive control (MPC) and existing VLM-integrated controllers, highlighting the effectiveness of context adaptation in enhancing driving safety and performance.
Anticipatory driving capabilities in autonomous vehicles can now be achieved without compromising safety, thanks to a new framework that adapts to real-time road conditions.
Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation, reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A vision-language model (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.
