Search papers, labs, and topics across Lattice.
This paper leverages the framework of actual causality to explain failures in autonomous Cyber-Physical Systems (CPSs), addressing the limitations of previous methods that focused on simpler systems. By developing two novel, system-agnostic algorithms, the authors enable the prioritization of either explanation optimality or derivation efficiency, significantly enhancing the understanding of unexpected system behaviors. The approach is validated through experiments with a neural-network-controlled autonomous vehicle, demonstrating its practical applicability in real-world scenarios.
Actual causality can now illuminate the black-box failures of autonomous systems, providing critical insights for improving reliability and trust.
Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures-when the system does not comply with its specification-are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions.
