Search papers, labs, and topics across Lattice.
This paper identifies a critical failure in multi-agent reinforcement learning (RL) systems that rely on a single large language model (LLM) for simulating user behavior, termed "simulator collapse." The authors theoretically formalize this collapse and propose two solutions: Verbalized Sampling at inference time, which broadens the simulator's behavior, and Co-Training at training time, which optimizes policies against a population of simulators. Experimental results demonstrate that these methods significantly improve generalization, with Verbalized Sampling enhancing success rates by up to 9% and Co-Training achieving up to 14% on various benchmarks, while also maintaining policy diversity.
Relying on a single simulator in multi-agent RL leads to dangerous mode collapse, but innovative solutions can boost generalization and performance by up to 14%.
Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.
BAIR
UW