Search papers, labs, and topics across Lattice.
The paper introduces COSMIC, a benchmark to evaluate MLLMs' ability to integrate partial spatial views through natural language dialogue, mimicking human collaborative spatial understanding. They test MLLMs on tasks requiring spatial reasoning and map building based on communication between two agents with different viewpoints in a 3D environment. Results show MLLMs struggle to build globally consistent maps and fail to converge on a shared understanding compared to humans, despite achieving reasonable accuracy on simpler tasks like identifying shared objects.
Even the best MLLMs still fall far short of human performance in collaborative spatial reasoning, revealing a critical gap in their ability to build and maintain shared mental models through dialogue.
Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce COSMIC, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. COSMIC contains 899 diverse scenes and 1250 question-answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human-human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue. Our code and data is available at https://github.com/ankursikarwar/Cosmic.
Mila