Search papers, labs, and topics across Lattice.
This paper introduces a dynamic manipulation hypergraph framework for human activity recognition that captures complex interactions among hands, objects, tools, and surfaces, addressing the limitations of traditional pairwise graph methods. By encoding features such as appearance, spatial, motion, and semantic roles at each temporal step, the framework enables a more nuanced understanding of multi-entity configurations. The proposed method significantly outperforms conventional approaches, achieving improvements in HO-F1 scores of up to 9.5 percentage points on benchmark datasets, highlighting the effectiveness of higher-order relational modeling in fine-grained manipulation recognition.
Dynamic manipulation hypergraphs reveal that recognizing multi-entity interactions can boost activity recognition performance by nearly 10 percentage points over traditional methods.
Fine-grained manipulation recognition requires modeling evolving relations among hands, objects, tools, and supporting surfaces. Conventional graph-based methods use pairwise edges that can fragment a coordinated event into disconnected binary relations. We propose a dynamic manipulation hypergraph framework that represents multi-entity configurations as higher-order relational units. At each temporal step, relevant entities are encoded using appearance, spatial, motion, and semantic-role features. Hyperedge candidates are instantiated and ranked using proximity, contact, and motion-coupling predicates. A hypergraph reasoning network performs node-to-hyperedge and hyperedge-to-node message passing, followed by temporal attention over the evolving interaction structure. The framework provides class-agnostic hyperedge-importance scores that identify entity configurations and temporal intervals emphasized by the model without treating them as causal explanations. Quantitative evaluation is conducted on EPIC-KITCHENS-100/VISOR and Assembly101 under an annotation-assisted entity-localization protocol. Video-only and entity-based methods provide contextual comparisons, while a matched pairwise graph and a static hypergraph serve as the principal controlled baselines because they use identical entity inputs and comparable relational settings. The proposed method improves HO-F1 over the matched pairwise graph by 6.9 percentage points on EPIC-KITCHENS-100/VISOR and 9.5 points on Assembly101, and exceeds the static hypergraph by 4.4 and 5.8 points, respectively. Qualitative analysis on ARCTIC further shows correspondence between highly ranked hyperedges and contact-rich manipulation intervals. These results demonstrate the value of time-varying higher-order relational modeling for fine-grained manipulation activity recognition.
