Search papers, labs, and topics across Lattice.
This paper introduces ESTANet, a lightweight framework for real-time online error detection in procedural videos that leverages the inherent prediction inconsistencies of action detectors. By analyzing the differing responses of standard and error-sensitive detectors to input dynamics, the authors demonstrate that these inconsistencies can effectively flag errors during procedural tasks. Extensive evaluations on multiple datasets reveal that ESTANet achieves state-of-the-art performance while maintaining real-time efficiency, underscoring the potential of utilizing existing action detector properties for practical applications.
Action detectors can reveal critical errors in procedural videos by exploiting their own prediction inconsistencies, achieving state-of-the-art performance without complex architectures.
An efficient and accurate system for detecting errors in procedural tasks is crucial for supporting human needs in daily life, as it can provide instant notifications and guide people to correct mistakes. In this work, we study real-time online error detection in procedural videos from a simple but overlooked perspective: the prediction behavior of action detectors themselves. Instead of designing complex architectures or specialized supervision, we observe that action detectors naturally exhibit different prediction characteristics depending on their sensitivity to input dynamics and temporal context. We therefore propose ESTANet (Error-Sensitive and Temporally-vArying Network), a lightweight framework that detects errors by exploiting inconsistencies among action predictions produced by a small set of action detectors. We construct standard and error-sensitive action detectors that behave similarly on correct executions but respond differently when errors occur. Meanwhile, detectors operating with different temporal contexts further amplify prediction inconsistencies when the procedure deviates from the intended sequence. During inference, we detect errors by aggregating mismatches between standard and error-sensitive predictions through majority voting to flag frames that contain errors. Extensive experiments on EgoPER, Assembly-101-O, and EPIC-Tent-O demonstrate that ESTANet achieves state-of-the-art performance in online error detection while maintaining real-time efficiency with a lightweight architecture. Our results highlight that leveraging the intrinsic properties of action detectors can yield a powerful and practical solution for online error detection without increasing architectural design complexity.
