Search papers, labs, and topics across Lattice.
This paper introduces NashDreamer, a centralized model-based reinforcement learning framework specifically designed for two-player zero-sum imperfect-information games (IIGs). By employing a Multi-Agent Recurrent State-Space Model (MARSSM), NashDreamer effectively separates environment dynamics from players' strategies, addressing the challenges of opponent-induced non-stationarity and identifiability in decentralized learning. Empirical results show that NashDreamer significantly enhances sample efficiency compared to model-free baselines, particularly in the early stages of training, while also revealing vulnerabilities in optimization landscapes for related algorithms.
NashDreamer achieves unprecedented sample efficiency in zero-sum imperfect-information games by centralizing model learning, challenging the status quo of decentralized approaches.
Model-based reinforcement learning (MBRL) has achieved remarkable results in single-agent domains, yet its extension to competitive imperfect information games (IIGs) remains underexplored. In multi-agent settings, opponent-induced non-stationarity complicates the learning process, and decentralized model learning faces severe identifiability barriers, which we argue make centralized model learning a mathematical necessity. Building on this analysis, we propose NashDreamer, a principled MBRL framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent State-Space Model (MARSSM) that decouples environment dynamics from the effect of players' strategies on their individual observations. NashDreamer is designed to use arbitrary policy gradient algorithms and inherits their convergence guarantees towards Nash equilibria under an idealized model. Empirical evaluations across four benchmark games demonstrate that NashDreamer substantially improves sample efficiency over model-free baselines early in the training. Finally, we theoretically analyze the architecture's optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.
