Search papers, labs, and topics across Lattice.
This study investigates the robustness of Autonomous Driving Systems (ADS) through real-world perturbation testing, addressing the limitations of previous research that relied heavily on offline datasets and simulations. By evaluating 72 camera and LiDAR perturbations across model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing, the authors reveal significant discrepancies in performance and failure prediction between these modalities. The findings indicate that model-level metrics are inadequate for identifying critical perturbations, emphasizing the necessity for real-world evaluations to ensure ADS reliability under diverse conditions.
Real-world testing uncovers that model-level metrics can mislead safety assessments, with camera systems exhibiting failures that offline evaluations fail to predict.
Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end vision-based driving model and a modular LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.
