Search papers, labs, and topics across Lattice.
The paper introduces WatchAct, a benchmark designed to evaluate robot manipulation grounded in observed human behavior by pairing real-world human-action videos with language instructions and aligned simulator scenes. This benchmark encompasses 3,000 long-horizon instances across 14 tasks, focusing on cognitive demands such as event grounding, procedural reasoning, implicit intent inference, and episodic reasoning. Current systems, including the Gemini-3.1-Pro pipeline, show limited success rates, achieving only 16.3% in simulation and 14.0% on a physical robot, highlighting the challenges in behavior-grounded manipulation tasks.
Robots struggle to match human-level reasoning in manipulation tasks, achieving only 16.3% success in a new benchmark that evaluates behavior-grounded reasoning.
A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that language leaves underspecified, yet today's manipulation benchmarks pair an instruction with a single current image, offering no way to evaluate reasoning over observed human behavior. We introduce WatchAct, a benchmark for robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another agent: parsing events (Event Grounding), recovering procedural structure (Procedural Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~video-to-plan reasoning by vision-language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner--policy pipelines. In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $蟺_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $蟺_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios. Dataset and code are available at https://baiqi-li.github.io/watchact_page/.
NVIDIA