Search papers, labs, and topics across Lattice.
The paper introduces Recurrent Reasoning Vision-Language Model ($\text{R}^2$VLM) for estimating task progress in long-horizon embodied tasks by iteratively processing local video snippets and maintaining a global context via an evolving Chain of Thought (CoT). This CoT explicitly records task decomposition, key steps, and completion status, enabling reasoning about complex temporal dependencies while avoiding the computational cost of processing long videos. Experiments on ALFRED and Ego4D datasets demonstrate $\text{R}^2$VLM achieves state-of-the-art performance in long-horizon task progress estimation and improves downstream applications like policy learning and reward modeling.
By iteratively reasoning over video snippets with a Chain-of-Thought, $\text{R}^2$VLM achieves state-of-the-art long-horizon task progress estimation without needing to process entire videos at once.
Accurately estimating task progress is critical for embodied agents to plan and execute long-horizon, multi-step tasks. Despite promising advances, existing Vision-Language Models (VLMs) based methods primarily leverage their video understanding capabilities, while neglecting their complex reasoning potential. Furthermore, processing long video trajectories with VLMs is computationally prohibitive for real-world deployment. To address these challenges, we propose the Recurrent Reasoning Vision-Language Model ($\text{R}^2$VLM). Our model features a recurrent reasoning framework that processes local video snippets iteratively, maintaining a global context through an evolving Chain of Thought (CoT). This CoT explicitly records task decomposition, key steps, and their completion status, enabling the model to reason about complex temporal dependencies. This design avoids the high cost of processing long videos while preserving essential reasoning capabilities. We train $\text{R}^2$VLM on large-scale, automatically generated datasets from ALFRED and Ego4D. Extensive experiments on progress estimation and downstream applications, including progress-enhanced policy learning, reward modeling for reinforcement learning, and proactive assistance, demonstrate that $\text{R}^2$VLM achieves strong performance and generalization, achieving a new state-of-the-art in long-horizon task progress estimation. The models and benchmarks are publicly available at \href{https://huggingface.co/collections/zhangyuelin/r2vlm}{huggingface}.
Tsinghua AI