Search papers, labs, and topics across Lattice.
This paper introduces a novel Hand-informed Context Enhanced (HiCE) module that uses cross-attention to learn spatiotemporal features from hand regions and their context for precise hand touch moment detection in egocentric video. To improve detection accuracy, they incorporate a grasp-aware loss and soft label to emphasize hand pose and movement dynamics indicative of touch. They also present TouchMoment, a new egocentric video dataset with 8,456 annotated contact moments, and demonstrate a 16.91% average precision improvement over state-of-the-art event spotting baselines under a strict two-frame tolerance evaluation.
Detecting the precise moment of hand-object contact in egocentric video is now significantly more accurate, thanks to a new module that focuses on hand pose and movement dynamics.
We address the challenging task of detecting the precise moment when hands make contact with objects in egocentric videos. This frame-level detection is crucial for augmented reality, human-computer interaction, assistive technologies, and robot learning applications, where contact onset signals action initiation or completion. Temporally precise detection is particularly challenging due to subtle hand motion variations near contact, frequent occlusions, fine-grained manipulation patterns, and the inherent motion dynamics of first-person perspectives. To tackle these challenges, we propose a Hand-informed Context Enhanced module (HiCE; pronounced `high-see') that leverages spatiotemporal features from hand regions and their surrounding context through cross-attention mechanisms, learning to identify potential contact patterns. Our approach is further refined with a grasp-aware loss and soft label that emphasizes hand pose patterns and movement dynamics characteristic of touch events, enabling the model to distinguish between near-contact and actual contact frames. We also introduce TouchMoment, an egocentric dataset containing 4,021 videos and 8,456 annotated contact moments spanning over one million frames. Experiments on TouchMoment show that, under a strict evaluation criterion that counts a prediction as correct only if it falls within a two-frame tolerance of the ground-truth moment, our method achieves substantial gains and outperforms state-of-the-art event-spotting baselines by 16.91% average precision.
