Search papers, labs, and topics across Lattice.
This paper introduces Inter-X++, a large-scale benchmark for analyzing multimodal human-human interactions, addressing limitations in existing datasets related to low-fidelity kinematics and lack of rich annotations. The dataset comprises 11,388 high-fidelity interaction sequences with over 8.1 million frames, featuring detailed whole-body movements and finger articulations, enriched with multifaceted annotations for comprehensive analysis. The authors also propose OpenHHI, a unified framework that optimizes interaction reconstruction and semantic understanding, achieving state-of-the-art performance across various generative and perceptive tasks.
A unified framework that simultaneously enhances interaction understanding and generation, achieving state-of-the-art results in multimodal human-human interaction analysis.
The capability to perceive and synthesize human-human interactions is fundamental to developing intelligent digital human systems. However, existing datasets and modeling approaches are fundamentally constrained by low-fidelity kinematics, the omission of dexterous hand gestures and a severe lack of rich multimodal annotations. Furthermore, fragmented interaction representations and inconsistent evaluation protocols also impede fair and rigorous benchmarking. To systematically address these bottlenecks, we present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis. Captured via a novel hybrid motion capture system, Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations. Meanwhile, we enrich the data foundation with multifaceted annotations, including hierarchical fine-grained textual descriptions, interaction categories, causal interaction orders, the relationship and personality of the subjects, as well as vertex-level contact maps and physically regularized constraints. Leveraging these elaborate annotations, we formulate a unified testing ground comprising four categories of downstream tasks that symmetrically span both generative and perceptive paradigms. To eliminate benchmarking ambiguities, we systematically standardize the interaction representations and evaluation protocols. Finally, we go beyond dataset construction to propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimizes interaction reconstruction and semantic understanding. Extensive experiments reveal that OpenHHI achieves state-of-the-art performance on both generation and perception tasks. This definitively proves that our unified representation successfully bridges interaction understanding and generation simultaneously.
