Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a pipeline to improve MLLM performance in identifying and reasoning about safety-critical events (SCEs) in driving videos. The pipeline fuses video frames with high-frequency telematics and semantic insights from CV models to generate high-quality pseudo-labels for fine-tuning. Fine-tuning QwenVL-2.5 with DoRA adapters on these pseudo-labels significantly improves SCE identification and explanation with limited computational resources.
MLLMs can learn to identify and explain safety-critical driving events with significantly improved accuracy by fusing multimodal sensor data and fine-tuning on targeted pseudo-labels, all while using fewer than 50M trainable parameters.
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding. However, their application to safety-critical driving scenarios remains limited by an inability to accurately perceive and reason about rare high-stakes dynamic events, such as collisions or near-collisions. To address this, we introduce a pipeline that enhances MLLM perception by fusing downsampled video frames with synchronized high-frequency telematics data (IMU and GPS) and semantic insights from specialized computer vision models. Our pipeline generates high-quality pseudo-labels, including descriptive captions and question-answer pairs, specifically designed to train MLLMs to identify and describe Safety-Critical Events (SCEs) in real-world driving footage. We show the effectiveness of our approach fine-tuning the open-source QwenVL-2.5 model via DoRA adapters: our experiments demonstrate significant improvements in identifying and explaining safety-critical events, with fewer than 50M trainable parameters and limited computational budget.