Search papers, labs, and topics across Lattice.
This study investigates the impact of supervised fine-tuning (SFT) on behavioral diversity in large language models (LLMs) during sequential decision-making tasks, specifically in deterministic board games derived from tic-tac-toe. The researchers found that while SFT improves action accuracy, it often leads to a significant reduction in action diversity, termed "diversity collapse," which can hinder exploratory behavior. By implementing action augmentation鈥攖raining on all optimal actions instead of a single demonstrated action鈥攖he authors demonstrate a method to partially counteract this detrimental effect, highlighting the importance of maintaining action diversity during model training.
SFT can lead to a drastic reduction in action diversity in LLMs, risking premature policy collapse even as accuracy improves.
Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks, but its effect on behavioral diversity in sequential decision-making remains under-explored. We study this question in a controlled suite of deterministic board games based on tic-tac-toe variants, where optimal actions are exactly computable and diversity can be measured directly. Across state-level evaluation, arena gameplay, and training trajectories, we find that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy. Furthermore, standard SFT improves accuracy but often induces premature diversity collapse, which exceeds what is minimally required by the accuracy-diversity tradeoff. We then show that action augmentation, which trains on all optimal actions per state rather than a single demonstrated action, would partially mitigates this effect. Our results identify narrow-support imitation as a source of policy collapse in LLM decision-making and suggest that preserving action support during SFT is important for maintaining exploratory behavior.
