Search papers, labs, and topics across Lattice.
This paper introduces VLAff, a vision-language-affordance model that learns actionable affordances from egocentric human videos to bridge the embodiment gap between humans and robots. By constructing the EgoAffordance dataset, which includes over 5.6 million visual affordances and 11.6 million grasp and trajectory affordances, the authors enable the model to generate actionable insights for robotic manipulation tasks. The results show that VLAff achieves state-of-the-art performance in visual affordance prediction and demonstrates practical applicability in zero-shot manipulation scenarios.
VLAff transforms human video insights into actionable robot manipulation skills, achieving state-of-the-art performance while enabling zero-shot learning capabilities.
Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real robot applications such as zero-shot manipulation and affordance-guided robot learning.
