Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper reframes post-training methods like SFT, RL, and on-policy distillation (OPD) for LLMs as state-distribution shaping processes, rather than focusing solely on loss functions. Through controlled experiments with Qwen3-0.6B-Base on GSM8K, the authors demonstrate that the distribution of states encountered during training significantly impacts both task performance and retention. Key findings include that OPD can outperform its SFT teacher despite using it as the sole supervision source, and that even mild SFT can cause substantial retention loss if the state distribution is not carefully managed.
The secret to successful LLM post-training isn't just *what* you train on, but *where* you train: the distribution of states encountered during training matters just as much as the supervision signal.
Large language model post-training methods such as supervised fine-tuning (SFT), reinforcement learning (RL), and distillation are often analyzed through their loss functions: maximum likelihood, policy gradients, forward KL, reverse KL, or related objective-level variants. We study a complementary factor: the state distribution on which supervision is applied. For an autoregressive policy, a state is a prompt plus generated prefix. SFT trains on fixed dataset states, while RL and on-policy distillation (OPD) train on states induced by the current learner. We formalize post-training as state-distribution shaping and run a controlled smallscale study using Qwen3-0.6B-Base on GSM8K, with TruthfulQA and MMLU as retention evaluations. Our results show three phenomena. First, a mild SFT run improves GSM8K with little forgetting, while a stress SFT run causes substantial retention loss. Second, OPD from a degraded SFT teacher surpasses that teacher on GSM8K, TruthfulQA, and MMLU, despite using the teacher as its only supervision source. Third, a lightweight on-policy RL run improves GSM8K while preserving retention. These results support a state-centric view of post-training: the source and locality of training states can be as important as the form of the supervision signal.
Meta AI