Search papers, labs, and topics across Lattice.
This paper introduces Regret-Guided Search Control (RGSC), a novel extension to AlphaZero that improves learning efficiency by prioritizing states with high regret, defined as the divergence between the agent's evaluation and the actual outcome. RGSC trains a regret network to identify these high-regret states from both self-play trajectories and MCTS nodes, storing them in a prioritized buffer for reuse as starting positions. Experiments across three games (9x9 Go, 10x10 Othello, and 11x11 Hex) demonstrate that RGSC significantly outperforms AlphaZero and Go-Exploit, achieving an average Elo improvement of 77 and 89, respectively, and further improving the win rate against KataGo when fine-tuning a pre-trained 9x9 Go model.
AlphaZero learns way faster when it focuses on replaying the moments it *almost* got right, improving by an average of 83 Elo across games.
Reinforcement learning (RL) agents achieve remarkable performance but remain far less learning-efficient than humans. While RL agents require extensive self-play games to extract useful signals, humans often need only a few games, improving rapidly by repeatedly revisiting states where mistakes occurred. This idea, known as search control, aims to restart from valuable states rather than always from the initial state. In AlphaZero, prior work Go-Exploit applies this idea by sampling past states from self-play or search trees, but it treats all states equally, regardless of their learning potential. We propose Regret-Guided Search Control (RGSC), which extends AlphaZero with a regret network that learns to identify high-regret states, where the agent's evaluation diverges most from the actual outcome. These states are collected from both self-play trajectories and MCTS nodes, stored in a prioritized regret buffer, and reused as new starting positions. Across 9x9 Go, 10x10 Othello, and 11x11 Hex, RGSC outperforms AlphaZero and Go-Exploit by an average of 77 and 89 Elo, respectively. When training on a well-trained 9x9 Go model, RGSC further improves the win rate against KataGo from 69.3% to 78.2%, while both baselines show no improvement. These results demonstrate that RGSC provides an effective mechanism for search control, improving both efficiency and robustness of AlphaZero training. Our code is available at https://rlg.iis.sinica.edu.tw/papers/rgsc.
