Search papers, labs, and topics across Lattice.
Pi-HOC, a novel framework, addresses the challenge of dense 3D semantic contact prediction in multi-human, multi-object interaction scenarios by generating human-object tokens and refining them with an InteractionFormer. This approach enables instance-aware contact estimation without requiring object geometries, unlike previous methods. Experiments on MMHOI and DAMON datasets show Pi-HOC achieves state-of-the-art accuracy and localization with a 20x speedup, and the predicted contacts improve SAM-3D image-to-mesh reconstruction.
Unlock 20x faster and more accurate 3D human-object contact estimation in complex, multi-person scenes with Pi-HOC, a framework that doesn't require object meshes.
Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.
CMU ML