Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces JOIN, a novel bimanual assistive manipulation system that combines a wheelchair-mounted anchor arm with a mobile manipulator to enhance the execution of complex daily tasks. By formulating the challenge of bimanual joining into a three-phase process—planning, driving, and grasping—JOIN effectively utilizes a vision-language model to coordinate the actions of both arms based on task-specific conditions. The results demonstrate that JOIN significantly outperforms existing methods, achieving 19 successful attempts out of 20 compared to 14 out of 20 for state-of-the-art systems, while also requiring less operator intervention.
JOIN achieves a remarkable 95% success rate in bimanual tasks, outperforming current systems and reducing operator corrections.
Assistive mobility and manipulation platforms have received increasing attention as a means of restoring independence to individuals with disabilities. While effective for many basic activities of daily living (ADLs), a significant percentage of everyday tasks such as opening a jar, pouring a liquid, lifting a tray, or basic meal preparation, is fundamentally bimanual and remains out of reach for any single-arm system. Adding a second arm to a wheelchair is impractical, due to the additional power draw, cost, and the loss of space required for transfers and mobility. We instead propose a heterogeneous, on-demand bimanual system, in which a wheelchair-mounted anchor arm is joined when needed by a summoned mobile manipulator that serves as a complement arm. The central technical problem, which we call bimanual joining, is conditional: the anchor has already committed to a grasp, and the complement arm must choose where to stand and what to grasp to complete the task. We formulate bimanual joining as a three-phase decomposition (plan, drive, grasp) and show that a vision-language model (VLM), coupled with standard geometric tools, provides task-level knowledge sufficient to solve a representative class of bimanual ADLs. Our system JOIN, contributes (i) a wheelchair-referenced opposition score, and (ii) task-conditioned directional manipulability. We evaluate JOIN on a Kinova Gen3 anchor and a Hello Robot Stretch~3 complement on representative same-object and different-object tasks. JOIN accomplished more attempts (19/20) than state-of-the-art methods (14/20) and required markedly less correction by the operator.