Search papers, labs, and topics across Lattice.
This study investigates the current practices and challenges in testing autonomous driving systems (ADS) through interviews with experts from nine companies across six countries. It highlights a focus on scenario-based and X-in-the-loop testing, while identifying major issues such as scenario realism and acceptance criteria. The research culminates in the proposal of an evidence-centered closed-loop testing framework aimed at enhancing the effectiveness and transparency of ADS testing practices.
Current ADS testing practices are hampered by major challenges, but an evidence-centered closed-loop framework could revolutionize how we ensure their safety and functionality.
Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.
