Search papers, labs, and topics across Lattice.
This paper introduces CoMind, a comprehensive dataset designed to study collaborative human activities through multiple perspectives in real-world cooking scenarios. By integrating egocentric and exocentric video, audio, gaze tracking, and 3D object scans, the dataset enables the exploration of cognitive processes like Theory of Mind in natural settings. Key benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction are established, paving the way for advancements in AI systems that understand and predict human collaborative behaviors.
Real-world collaboration can now be analyzed through a rich, multimodal lens that captures the intricacies of human interaction in cooking tasks.
Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.
