Search papers, labs, and topics across Lattice.
RoboWits, a new bi-manual robotic benchmark, is introduced to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. The benchmark uses an automated task generation pipeline formulated as a multi-agent cooperative framework to create diverse seed tasks and mutated tasks with graded difficulty. Benchmarking results show that while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle on mutated tasks, highlighting their brittleness in manipulation tasks requiring reasoning and adaptation.
Today's robot policies and VLAs fall apart when faced with unexpected challenges requiring reasoning, strategy adaptation, and robustness, even after fine-tuning on similar tasks.
The ability to reason, adapt, and creatively solve problems under unexpected challenges is essential for robots operating in real-world environments. However, current robotic benchmarks primarily emphasize skill-level execution and provide limited insight into such cognitive reasoning capabilities. We introduce RoboWits, a bi-manual robotic benchmark designed to systematically evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. To enable scalable construction of high-quality reasoning-centric unexpected scenarios, we propose an automated task generation pipeline formulated as a multi-agent cooperative framework, comprising agents for seed task generation and verification, metric generation, scene generation, and task mutation. Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning. We benchmark popular robot policies, pre-trained VLAs, and oracle-state planners. Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks, implying their brittleness in manipulation tasks requiring reasoning, strategy adaptation, and robustness to deceptive or constrained environments. Project page is available at https://umass-embodied-agi.github.io/RoboWits.
AI2