Search papers, labs, and topics across Lattice.
This paper introduces LUGL (Local Updates, Global Learning), a novel framework that allows gradient-boosted trees (GBTs) to effectively operate in reinforcement learning (RL) settings by decoupling data collection from model fitting. By alternating between local updates during self-play and global learning to generalize from accumulated data, LUGL enables GBTs to overcome challenges posed by distributional shifts in game environments. The results indicate that GBT-based agents can achieve performance on par with or better than traditional deep learning approaches like DQN and DeepCFR across various perfect and imperfect information games, challenging the prevailing preference for neural networks in game-playing contexts.
GBTs can outperform neural networks in game-playing scenarios, revealing a potential misalignment in the community's reliance on deep learning methods.
The dominance of Neural Networks (NNs) in RL is partially due to their incremental learning capability, which naturally suits the online, non-stationary nature of self-play training. However, gradient-boosted trees like LightGBM are widely recognised as the state of the art for tabular data in supervised learning, often outperforming NNs in accuracy and efficiency. Game states are inherently tabular---discrete actions, categorical card identities, structured board positions---which makes them an ideal candidate for tree-based methods. We introduce LUGL (Local Updates, Global Learning), a framework that decouples data collection from model fitting, enabling non-incremental learners such as GBTs to operate in RL settings where they would otherwise fail due to distributional shift. LUGL alternates between a local updates phase, where the agent plays self-play games and accumulates tabular updates (Q-values, V-values, policies, or regret values) in a finite table, and a global learning phase, where the table is used to train a function approximator that generalises to unseen states before the table is reset. We test our approach in four standard perfect-information games (Tic-tac-toe, Connect-4, Othello, and Hex) and five imperfect-information games (Kuhn's poker, Leduc Hold'em, Liar's Dice, Goofspiel, and Flop5 Hold'em), and show that our results are competitive with or superior to DQN and DeepCFR. Our experiments demonstrate that the community's strong bias towards NNs in game-playing may be unwarranted, since LightGBM-based agents achieve competitive or superior performance across all tested benchmarks.
