Search papers, labs, and topics across Lattice.
The paper introduces REACT, a V2X-integrated trajectory optimization framework for autonomous driving that leverages a fine-tuned lightweight Vision-Language Model (VLM) to fuse infrastructure-provided hazard alerts with onboard sensor data. REACT employs Residual Trajectory Fusion (RTF) and edge-adaptation strategies to reduce model complexity and improve inference efficiency for real-time deployment. Evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance with a 77% collision rate reduction and a 0.57-second inference latency on the Jetson AGX Orin, demonstrating the effectiveness of lightweight VLMs for cooperative planning.
Autonomous vehicles can now react to hazards in under a second thanks to a lightweight vision-language model that fuses onboard sensors with infrastructure alerts.
Autonomous driving (AD) systems relying solely on onboard sensors may fail to detect distant or obstacle hazards, potentially causing preventable collisions; however, existing transformer-based Vehicle-to-Everything (V2X) approaches, which mitigate AD sensing limitations, either lack effective multimodal fusion and reasoning or struggle to meet real-time performance requirements under complex, high-dimensional traffic conditions. This paper proposes the Real-time Edge-based Autonomous Co-pilot Trajectory planner (REACT), a V2X-integrated trajectory optimization framework for AD based on a fine-tuned lightweight Vision-Language Model (VLM). REACT integrates infrastructure-provided hazard alerts with onboard sensor data, capturing intricate surrounding traffic dynamics and vehicle intents through visual embeddings, interpreting precise numerical data from symbolic inputs, and employing contextual reasoning to generate optimized, safety-oriented trajectories. To ensure robust real-time deployment on edge devices, REACT innovatively employs Residual Trajectory Fusion (RTF) design and specialized edge-adaptation strategies to reduce model complexity and improve inference efficiency. Evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance, a 77% collision rate reduction, a 48.2% Video Panoptic Quality (VPQ), and a 0.57-second inference latency on the Jetson AGX Orin. Ablation studies validate the contribution of each input, module, and edge adaptation strategy. These results highlight the effectiveness of lightweight VLMs in enabling real-time cooperative planning on edge platforms and underscore the potential of language-guided contextual reasoning for improving traffic safety and responsiveness.
