Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors investigate how supervised fine-tuning (SFT) and reinforcement learning (RL) impact language model reasoning in chess, using theoretically-inspired datasets. They find that while direct move prediction via SFT leads to effective RL and strong performance, it also results in unfaithful reasoning, whereas training on multi-move trajectories yields comparable performance with more faithful reasoning and stable RL. Furthermore, RL significantly improves move quality distribution and reduces hallucination, and certain SFT metrics are predictive of post-RL performance.
Training LMs on multi-move trajectories achieves comparable chess performance to direct move prediction, but with more faithful reasoning and stable RL – suggesting a critical trade-off between optimization efficiency and reasoning integrity.
How can you get a language model to reason in a task it natively struggles with? We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets impacts language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL step elicits unfaithful reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We show that RL induces a substantial positive shift in the distribution of move quality and reduces hallucination rates as a side effect. Finally, we find several SFT-checkpoint metrics -- metrics spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. We release checkpoints and final models as well as training data, evaluations, and code which allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model.