Search papers, labs, and topics across Lattice.
This paper addresses the limitations of current evaluation methods for autonomous driving in long-tail scenarios by introducing FluidTest, a comprehensive evaluation pipeline that combines human annotation, threat taxonomy, and a verification system. The study reveals that even advanced planners can exhibit significant safety failures, as evidenced by FluidTest identifying new threats in over half of the evaluated trajectories despite high performance metrics. By enhancing the reliability and explainability of evaluation processes, this work aims to bridge the gap between automated assessments and human-aligned safety standards in autonomous driving.
Advanced planners still exhibit critical safety failures, with FluidTest uncovering new threats in over 65% of evaluated trajectories.
Long-tail scenarios remain a major bottleneck for autonomous driving evaluation, even as datasets grow by orders of magnitude. Existing evaluation pipelines are rarely human-aligned, safety-aware, verifiable, and explainable at the same time: closed-loop metrics often saturate among strong planners, while unstructured human ratings can be noisy without a carefully designed protocol. We formulate planning evaluation as additional-threat detection: given a planner trajectory and an expert reference, does the planner's displacement introduce new unsafe driving behavior? We propose FluidTest, an evaluation pipeline with three components: a pairwise WebUI protocol for reliable human annotation; a taxonomy of 32 semantic threats with evidence-grounded decision graphs; and a three-agent verification system with reflection for precision and auditability. Experiments on the WOD-E2E dataset show that FluidTest produces consistent labels among trained annotators and identifies additional threats in 65% of Poutine trajectories and 51% of RAP trajectories. These results show that state-of-the-art planners can still exhibit substantial safety-relevant failures despite high Rater Feedback Scores (RFS) and low Average Displacement Error (ADE). Additional details, guidance, and code are available at https://fluidtest.web.app.
Tsinghua AI