Search papers, labs, and topics across Lattice.
This paper investigates the correlation between offline and online evaluation metrics for autonomous driving policies, finding a surprisingly weak relationship that challenges current offline evaluation practices. They propose a novel offline metric based on epistemic uncertainty to better predict closed-loop performance. The proposed metric demonstrates a 13% improvement in correlation with online performance in simulation and even greater gains in real-world validation.
Offline metrics for autonomous driving are even more misleading than we thought, but a new epistemic uncertainty-based metric closes the gap by 13%.
Real-World evaluation of perception-based planning models for robotic systems, such as autonomous vehicles, can be safely and inexpensively conducted offline, ı.e., by computing model prediction error over a pre-collected validation dataset with ground-truth annotations. However, extrapolating from offline model performance to online settings remains a challenge. In these settings, seemingly minor errors can compound and result in test-time infractions or collisions. This relationship is understudied, particularly across diverse closed-loop metrics and complex urban maneuvers. In this work, we revisit this undervalued question in policy evaluation through an extensive set of experiments across diverse conditions and metrics. Based on analysis in simulation, we find an even worse correlation between offline and online settings than reported by prior studies, casting doubts on the validity of current evaluation practices and metrics for driving policies. Next, we bridge the gap between offline and online evaluation. We investigate an offline metric based on epistemic uncertainty, which aims to capture events that are likely to cause errors in closed-loop settings. The resulting metric achieves over 13% improvement in correlation compared to previous offline metrics. We further validate the generalization of our findings beyond the simulation environment in real-world settings, where even greater gains are observed.
