Search papers, labs, and topics across Lattice.
This paper analyzes the failure modes of two-stage Human-Object Interaction (HOI) detection models by decomposing HOI detection into interpretable perspectives such as multi-person interactions and object sharing. They curate a subset of an existing HOI dataset organized by human-object-interaction configurations to analyze model behavior under different scene compositions. The study reveals that high overall benchmark performance does not guarantee robust visual reasoning about human-object relationships, particularly in complex scenes.
HOI models ace benchmarks but still flunk basic visual reasoning when humans start sharing objects or interacting in groups.
Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area.
