Search papers, labs, and topics across Lattice.
This paper introduces a novel scenario generation pipeline that leverages historical failure records from Autonomous Driving Systems (ADS) to create diverse testing scenarios. By utilizing modular LLM-based synthetic scenario generation, the authors effectively transform natural language failure descriptions into structured scenarios that adhere to specific testing constraints. The approach not only enhances the diversity and accuracy of generated scenarios but also uncovers critical system failures within a constrained budget of just 20 scenarios, demonstrating its practical utility in pre-deployment testing.
Historical failure records can be transformed into diverse testing scenarios for autonomous driving, revealing critical system vulnerabilities with minimal effort.
To ensure safe on-road behavior, pre-deployment testing and failure discovery of Autonomous Driving Systems (ADS) is crucial. Present day simulation based testing methods focus largely on mathematical models for efficient search of optimal scenarios, assuming a fixed scenario representation. On the other hand, real-world testing involves substantial manual effort to design scenario templates for testing. These templates represent distinct failure scenarios consisting of pre-deployment vehicle movements, map types, etc. Historical failure records for ADS are a reliable source of real-world failure conditions, which can be used for scenario generation. In this work, we propose a scenario generation pipeline using categorical and contextual information available from historical records in natural language format. Our approach consists of modular LLM based synthetic scenario generation, compatible with the testing constraints of a given system. We successfully apply our method to generate a diverse set of scenarios for testing autonomous navigation on Metadrive simulator using the NHTSA ADS crash records. Our approach results in accurate and diverse scenario generation with a combination of 4 road types, 3 non ego vehicle movement types, including on road anomalies in the form of working zones. Generated scenarios align with the provided testing conditions, and reveals interesting failures of the system within a limited testing budget of 20 scenarios. Code is available at https://github.com/anjaliParashar/crash2scenario.
MIT CSAIL