Search papers, labs, and topics across Lattice.
This paper introduces EPIC-Contact, a novel dataset comprising 2.3K clips with dense 3D hand-object contact correspondences, addressing the challenges of estimating 3D hand-object poses in real-world scenarios plagued by occlusions and ambiguous interactions. To leverage this dataset, the authors present HOPformer, an end-to-end transformer model that predicts bi-manual hand and object poses simultaneously through a cross-attention mechanism that conditions object features on hand priors. The results demonstrate that HOPformer achieves an 82.4% success rate on the ARCTIC dataset, surpassing the previous state-of-the-art by 6.2 points, and nearly doubles the success rate on EPIC-Contact while significantly reducing contact deviation by 75%.
A new dataset and transformer model that together nearly double the success rate for 3D hand-object pose estimation in real-world settings.
Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.
