Search papers, labs, and topics across Lattice.
This paper introduces RoboVAST, a framework for automated scenario-based validation of robotic systems that addresses the challenges of manual scenario selection and variation. By modeling scenarios compositionally and enabling scalable execution, RoboVAST facilitates comprehensive testing across a vast array of configurations, demonstrated through extensive simulations involving over 100,000 runs and 5480 unique scenarios. The key finding reveals that systematic failures can be effectively distinguished from stochastic anomalies, enhancing the reliability of validation processes in robotics.
RoboVAST can execute over 100,000 validation runs, enabling the identification of systematic failures in robotic systems that were previously obscured by noise.
Validation of robotic systems critically depends on the operating conditions under which they are assessed. Scenario selection and variation are often manual, experience-driven, and difficult to scale, which harms reproducibility and weakens validation conclusions. We propose a scenario-based methodology that models scenarios compositionally and formalizes how these dimensions are varied, instantiated, executed, and interpreted. Building on this, we introduce RoboVAST, a framework that realizes declarative campaign specifications, plugin-based scenario generation, and scalable containerized execution with integrated result analysis. We demonstrate the approach with a navigation dataset comprising 5480 scenario configurations and over 100000 execution runs across five indoor maps with varied paths, sensor noise, software parameters, and obstacle settings, totaling more than 1800 hours of simulated operation and 1873 km traveled. Twenty repetitions per configuration allow us to distinguish systematic failures from stochastic anomalies.
