Search papers, labs, and topics across Lattice.
GraspDreamer is introduced, a novel approach that leverages visual generative models (VGMs) to synthesize human demonstrations for training robots to perform zero-shot functional grasping. By tapping into the implicit understanding of human-object interactions encoded within VGMs pre-trained on internet-scale data, GraspDreamer circumvents the need for extensive real-world data collection. Experiments on public benchmarks and real-world robots demonstrate GraspDreamer's superior data efficiency and generalization capabilities compared to existing methods, and its potential for downstream manipulation tasks and visuomotor policy learning.
Skip the costly real-world data collection: GraspDreamer uses generated human demonstrations from visual generative models to train robots for zero-shot functional grasping.
Building generalist robots capable of performing functional grasping in everyday, open-world environments remains a significant challenge due to the vast diversity of objects and tasks. Existing methods are either constrained to narrow object/task sets or rely on prohibitively large-scale data collection to capture real-world variability. In this work, we present an alternative approach, GraspDreamer, a method that leverages human demonstrations synthesized by visual generative models (VGMs) (e.g., video generation models) to enable zero-shot functional grasping without labor-intensive data collection. The key idea is that VGMs pre-trained on internet-scale human data implicitly encode generalized priors about how humans interact with the physical world, which can be combined with embodiment-specific action optimization to enable functional grasping with minimal effort. Extensive experiments on the public benchmarks with different robot hands demonstrate the superior data efficiency and generalization performance of GraspDreamer compared to previous methods. Real-world evaluations further validate the effectiveness on real robots. Additionally, we showcase that GraspDreamer can (1) be naturally extended to downstream manipulation tasks, and (2) can generate data to support visuomotor policy learning.
