Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DeepIPCv3, a multi-modal navigation framework that integrates LiDAR point clouds with asynchronous event streams from a Dynamic Vision Sensor (DVS) to enhance pedestrian crossing avoidance in autonomous vehicles. By employing a Transformer-inspired cross-modal attention mechanism, the system dynamically prioritizes critical updates while maintaining scene awareness, leading to improved trajectory and control command accuracy. Extensive offline evaluations reveal that DeepIPCv3 significantly reduces errors associated with motion blur and exposure failures, achieving state-of-the-art performance in diverse lighting conditions.
By fusing LiDAR and event-based vision, DeepIPCv3 enables autonomous vehicles to react to sudden pedestrian crossings with unprecedented accuracy and speed.
Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings. To address this critical safety vulnerability, we propose DeepIPCv3, a novel multi-modal autonomous navigation framework that synergizes the dense 3D spatial geometry of LiDAR point clouds with the microsecond-level asynchronous event streams of a Dynamic Vision Sensor (DVS). We introduce a Transformer-inspired cross-modal attention mechanism to dynamically correlate these distinct modalities, allowing the network to instantaneously prioritize high-speed dynamic updates without sacrificing structural scene awareness. The fused latent representations are then mapped to safe local waypoints and executable control commands via a hybrid policy network that blends heuristic trajectory tracking with direct neural predictions. Due to the severe physical risks associated with live testing of these sudden crossing scenarios, the framework is rigorously evaluated offline using a custom multi-modal dataset collected across both well-illuminated noon and challenging evening conditions. Extensive comparative and ablation studies demonstrate that DeepIPCv3 achieves state-of-the-art predictive performance. By effectively eliminating exposure failures and motion blur, the proposed LiDAR and DVS fusion yields the lowest trajectory and control command errors, enabling highly reactive, mathematically bounded evasive maneuvers regardless of ambient illumination. To support future research, we will release the codes to our GitHub repo at https://github.com/oskarnatan/DeepIPCv3.