Search papers, labs, and topics across Lattice.
This paper introduces EMAgnet, a novel regularization method for policy gradient self-play in large games, which adapts the regularization target to an exponential moving average of the last-iterate policy's parameters. By evolving the regularization strategy in tandem with the agent's learning, EMAgnet outperforms traditional uniform distribution regularization, particularly in environments with exploration challenges and dominated strategies. The results demonstrate that EMAgnet achieves lower exploitability than PPO with uniform-magnet regularization across various benchmarks, indicating its effectiveness in enhancing policy performance in complex game settings.
EMAgnet reduces exploitability in self-play scenarios by dynamically adapting regularization to the agent's evolving strategy, outperforming traditional methods.
Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy's parameters, providing an adaptive regularization target that evolves with the agent's improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies.
