Search papers, labs, and topics across Lattice.
This paper introduces CoDex, a zero-demonstration framework for learning Compositional Dexterous Functional Object Manipulation (CD-FOM) tasks, which involve both the actuation of an object's internal mechanism and precise control of its pose. By leveraging vision-language models to infer semantic constraints, CoDex employs analytic constrained optimization to generate functional grasp candidates, which are then refined through reinforcement learning. The framework is validated on a 7-DoF robot arm with a multi-fingered hand, successfully executing complex manipulations on previously unseen objects, demonstrating its capability to autonomously learn dexterous behaviors without prior demonstrations.
CoDex enables robots to autonomously master complex manipulation tasks without any human demonstrations, achieving dexterity previously thought unattainable.
In this work, we study Compositional Dexterous Functional Object Manipulation (CD-FOM): tasks such as aiming and actuating a spray bottle on a plant or a glue gun on wood, which require both actuating an object's internal mechanism and controlling its pose to apply the object's function to the environment. These tasks pose significant challenges for robots due to the demanding integration of semantic understanding of the object's function, actuation mode, and application area with intricate physical dexterity to manage grasp stability, movement trajectory, and actuation. We introduce CoDex, a zero-demonstration framework that autonomously discovers CD-FOM manipulation strategies. CoDex uses vision-language models (VLMs) to infer semantic constraints from the task and scene. These constraints guide analytic constrained optimization to generate a short list of functional grasp candidates that can be efficiently refined with reinforcement learning to generate full grasp-move-actuate policies transferable from simulation to the real world. We evaluate CoDex on a 7-DoF robot arm with a 16-DoF multi-fingered hand across six CD-FOM tasks involving previously unseen objects with internal mechanisms, including spray bottles, hot glue guns, air dusters, flashlights, and pepper grinders, and their application to unseen target objects, showcasing its ability to autonomously discover and execute complex, physically viable dexterous behaviors without human demonstrations. More information at https://robin-lab.cs.utexas.edu/CoDex/.
