Search papers, labs, and topics across Lattice.
This paper introduces HATS, a novel human-agent teleoperation system designed for efficient data collection in multi-arm manipulation tasks. By allowing a single human operator to control two primary robotic arms while an MLLM-based agent manages two assistive arms, HATS effectively balances cognitive load and coordination costs. Evaluations show that HATS achieves data collection efficiency and success rates on par with expert dual-human teams, highlighting its potential for training coordinated manipulation policies.
A single operator can now achieve multi-arm manipulation efficiency comparable to expert teams, thanks to HATS' innovative human-agent collaboration.
Many real-world manipulation scenarios, such as handling complex collaborative tasks and dealing with large workspaces, require coordination of more than two robotic arms. Consequently, an effective multi-arm teleoperation system is required to collect demonstrations for training coordinated multi-arm manipulation policies. However, existing teleoperation frameworks mainly focus on single-operator or multi-operator setups, facing a practical trade-off between the cognitive load placed on a single operator and the coordination cost incurred by multiple operators. To address this problem, we introduce HATS, a human-agent teleoperation system that enables a single human operator, assisted by an MLLM-based agent, to collect data for multi-arm manipulation tasks. Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free agent that handles sub-tasks. In addition, the human operator can use voice commands to prevent collisions and correct assistive arm behaviors during execution. Extensive evaluations demonstrate that HATS achieves data collection efficiency and success rates comparable to expert dual-human teams. Moreover, downstream policy evaluations demonstrate the efficacy and quality of the data collected through HATS.
