Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
To evaluate whether benchmark gains translate to genuine behavioral improvements when driving scores are used as optimization objectives, the authors decompose the autonomous driving evaluation pipeline into execution, measurement, subscore mapping, and aggregation. Controlled behavioral interventions reveal that significant driving deviations often trigger negligible score changes due to aggressive thresholding and attenuation between requested and executed trajectories. Crucially, closed-loop evaluations show that optimization gains on standard benchmarks can outright reverse when the low-level execution interface changes, exposing heavy reliance on simulator-specific feedback dynamics.
Optimizing autonomous driving policies directly against benchmark scores produces gains that can completely reverse under altered execution interfaces, revealing that top-performing models often exploit measurement artifacts rather than improving vehicle control.
Driving benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score gains remain reliable evidence of driving improvement once the score itself is optimized? We address this question by examining how the scoring process responds to changes in driving behavior and whether the resulting gains persist under repeated execution and replanning. We decompose the process into execution, measurement, subscore mapping, and aggregation. Controlled interventions reveal substantial behavioral changes that receive little score response because distinctions are omitted, thresholded, or attenuated between requested and executed motion. Closed-loop comparisons further show that optimization gains can reverse when the execution interface changes, demonstrating their dependence on how requests are executed and returned as feedback. Together, these findings connect the behavioral distinctions preserved by a metric to the conditions under which its gains transfer. Metric validity under optimization therefore requires examining both what the scoring process measures and how the optimized behavior is executed.