Search papers, labs, and topics across Lattice.
This paper introduces the EgoErrorVQA task to assess egocentric procedural comprehension in Visual Agents and Visual Language Models (VLMs) by focusing on their ability to detect procedural errors. The authors develop a user-friendly evaluator agent based on the Agent2Agent (A2A) protocol, which allows for standardized evaluation through visual question answering (VQA). Results show that existing models struggle with procedural error detection, but the proposed Adaptive Decoupled Reasoning (Ego-ADR) framework significantly improves performance, achieving state-of-the-art results on various metrics.
Visual agents consistently falter at detecting procedural errors, but a new framework shows how to enhance their comprehension capabilities dramatically.
The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents and Visual Language Models (VLMs) overlook the evaluation of their procedural comprehension ability from an egocentric visual perspective, particularly for detecting procedural errors, a critical capability for everyday assistance. To bridge this gap, the EgoErrorVQA task is firstly proposed for egocentric procedural comprehension with explicit procedural errors modeling. Besides, we develop a user-friendly evaluator agent based on the Agent2Agent (A2A) protocol, enabling rigorous and standardized evaluation of visual agents through VQA-based interaction. A range of models are evaluated using both open-ended and multiple-choice questions, revealing persistent weaknesses in handling procedural errors and error types. Moreover, we introduce Ego-ADR, an Adaptive Decoupled Reasoning framework that decouples complex procedural reasoning to enhance models'understanding of procedural errors. It achieves consistent performance gains over the selected baselines and attains state-of-the-art results on several metrics under comparable settings. Code: https://github.com/z1oong/EgoErrorVQA
