Search papers, labs, and topics across Lattice.
This paper introduces CollabBench, a novel benchmark designed to evaluate and enhance the collaborative abilities of LLM-based agents in cooperative game environments. By employing a Diverse Player Profile Simulation pipeline and a Collaborative Agentic Training paradigm, the study enables agents to effectively engage with varied human behaviors, resulting in significant improvements in both task efficiency and affective performance. Experimental results indicate that models trained with CollabBench achieve a 19.5% increase in efficiency and a 24.4% boost in affective performance compared to baseline models, highlighting existing collaborative limitations in current LLMs.
LLMs can significantly enhance their collaborative performance, achieving over 24% better engagement with human partners when trained in realistic game scenarios.
While LLM-based agents excel at individual tasks, effective collaboration with realistic human partners remains challenging. Most of the existing conversation-level collaborative studies lack grounded interaction and behavioral execution, motivating the need for cooperative game environments that enable contextualized and immersive collaboration. To this end, this paper proposes CollabBench, a benchmark for evaluating and training collaborative agents in cooperative games. CollabBench features a Diverse Player Profile Simulation pipeline to model varied players behaviors, and a Collaborative Agentic Training paradigm that unifies reasoning, communication, and action via agentic rollouts, optimized with a hybrid reward balancing task efficiency and affective adaptation. We further extend classic environments to CWAH-MultiPlayer and Cook-MultiPlayer for systematic evaluation under diverse personalities. Experiments with efficiency and affective metrics show that our trained models outperform base models, achieving 19.5% higher efficiency and 24.4% improved affective performance. Further analysis reveals key collaborative limitations of existing models and offers insights for future collaborative training.
