Search papers, labs, and topics across Lattice.
The paper introduces video-guided 3D affordance grounding, a novel approach that leverages dynamic human-object interaction (HOI) sequences in videos to identify regions on 3D objects that support interaction. This addresses the limitations of existing static-based methods that struggle to localize true contact regions. The authors propose VAGNet, a framework that aligns video-derived interaction cues with 3D structure, and introduce PVAD, a new HOI video-3D pairing affordance dataset, demonstrating state-of-the-art performance compared to static baselines.
Forget static cues: VAGNet grounds 3D object affordances by watching how humans actually use them in videos, significantly improving localization of interaction regions.
3D object affordance grounding aims to identify regions on 3D objects that support human-object interaction (HOI), a capability essential to embodied visual reasoning. However, most existing approaches rely on static visual or textual cues, neglecting that affordances are inherently defined by dynamic actions. As a result, they often struggle to localize the true contact regions involved in real interactions. We take a different perspective. Humans learn how to use objects by observing and imitating actions, not just by examining shapes. Motivated by this intuition, we introduce video-guided 3D affordance grounding, which leverages dynamic interaction sequences to provide functional supervision. To achieve this, we propose VAGNet, a framework that aligns video-derived interaction cues with 3D structure to resolve ambiguities that static cues cannot address. To support this new setting, we introduce PVAD, the first HOI video-3D pairing affordance dataset, providing functional supervision unavailable in prior works. Extensive experiments on PVAD show that VAGNet achieves state-of-the-art performance, significantly outperforming static-based baselines. The code and dataset will be open publicly.
Tsinghua AI