Search papers, labs, and topics across Lattice.
This paper introduces the EgoProceVQA task, which evaluates the procedural understanding capabilities of multi-modal large language models (MLLMs) through a novel set of key-step-centric questions. To facilitate this evaluation, the authors developed EgoProceGen, a data generation platform that creates tailored question-answer pairs for various procedural scenarios, resulting in a benchmark of 3,600 questions across 31 everyday tasks. The findings reveal that current MLLMs and agents exhibit significant deficiencies in procedural reasoning, prompting the introduction of EgoProceAgent, a self-skill-exploration framework that enhances performance through skill composition and selection strategies.
Existing MLLMs struggle with procedural reasoning, but a new self-skill-exploration agent achieves state-of-the-art performance by discovering effective strategies for everyday tasks.
Most daily activities are inherently procedural. However, existing evaluations for egocentric video understanding seldom address procedural understanding and largely overlook complex key-step-level reasoning under the widely used video question answering (VQA) paradigm for MLLMs. Such capabilities are crucial for building procedural AI assistants deployable on wearable devices. To bridge this gap, we introduce the Egocentric Procedural Understanding VQA task (EgoProceVQA), which systematically evaluates egocentric procedural reasoning abilities of current MLLMs and agents through six types of key-step-centric questions. Furthermore, we develop EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types. Based on this platform, we build a benchmark with 3,600 questions, four common procedural scenarios, and 31 everyday procedural tasks. Evaluations on EgoProceVQA show that existing MLLMs and agents still have substantial room for improvement in procedural understanding. Therefore, we further propose EgoProceAgent, a self-skill-exploration agentic framework. We design a generic tool library for procedural understanding and a standardized sub-skill library shared across tools and models, enabling self-exploration without ground-truth supervision. By exploring how to compose and select sub-skills, the agent discovers effective skill strategies for diverse problems, and attains state-of-the-art performance among open-source models on multiple tasks. Together, our benchmark, generation platform, and agentic framework establish a unified foundation for EgoProceVQA. Project page: https://z1oong.github.io/EgoProceVQA/.
