Search papers, labs, and topics across Lattice.
This paper introduces a zero-shot framework for long-horizon dexterous manipulation that translates language instructions into executable 3D task plans using calibrated multi-view RGB images. By leveraging a vision-language model (VLM) for reference-frame task grounding and 2D keypoint extraction, the system employs multi-view fusion techniques to achieve accurate 3D lifting of task-related keypoints. Experimental results indicate significant improvements in 3D grounding accuracy and execution reliability, enabling the system to perform complex manipulation tasks on unseen objects in novel environments.
Achieving reliable long-horizon manipulation without prior training on specific tasks, this framework enables robots to adaptively execute complex actions in new environments based solely on language instructions.
We present a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Rather than training an end-to-end policy, our system uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multi-view fusion. This lifting combines triangulation of view-wise VLM groundings with reference-view ray voting, which searches along a semantic camera ray for geometrically consistent candidates across neighboring views. The resulting 3D keypoints support both pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines. We further demonstrate long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes.
