Search papers, labs, and topics across Lattice.
This study introduces MECoBench, a comprehensive benchmark designed to evaluate multimodal large language models (MLLMs) in collaborative tasks within visually grounded environments. The research reveals that while collaboration enhances task completion, its effectiveness is contingent on the interplay between collaborative benefits and coordination complexity, with communication being a critical factor. Additionally, the findings indicate that collaboration not only improves performance but also enhances robustness in challenging conditions, providing valuable insights into the dynamics of multimodal agent cooperation.
Collaboration in multimodal tasks can significantly boost performance, but its success hinges on effective communication and the right team dynamics.
Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored. To address this gap, we introduce MECoBench, a multimodal embodied cooperation benchmark with an evaluation platform spanning diverse real-world tasks, two cooperation structures, and three collaboration modes. Through extensive experiments across various MLLMs, we summarize three key findings: (i) Collaboration generally improves embodied task completion, but its benefits depend on balancing collaborative gains against coordination complexity. (ii) Communication is essential to collaboration gains, while the best collaboration mode depends on team size and model capability. (iii) Moreover, collaboration improves robustness under noisy priors and exploration conditions. Generally, MECoBench provides a systematic testbed for understanding the mechanisms and limits of multimodal embodied collaboration. Code and dataset are available at https://github.com/q-i-n-g/MECoBench.
