Search papers, labs, and topics across Lattice.
This paper introduces a novel learning paradigm for hand-object interaction (HOI) recognition that integrates hand-object masked training with an HOI-dynamics-aware decoder, allowing models to reason more effectively from partial observations of hands and objects. By leveraging a new evaluation framework called Cue-Isolated HOI (CI-HOI) and the DEHOI testbed, the authors systematically assess the models' ability to independently predict actions based on hand- and object-related cues. The results show that their approach significantly outperforms existing models across various tasks, enhancing the robustness of HOI understanding.
Models trained with hand-object masked strategies can achieve superior HOI recognition by effectively isolating and utilizing hand- and object-centric cues.
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models'ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.
