Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper challenges the conventional wisdom that supervised finetuning (SFT) inherently suffers from poor generalization and catastrophic forgetting compared to reinforcement learning (RL) by introducing a novel Markov chain Monte Carlo (MCMC) sampling algorithm. This algorithm transforms off-policy expert data into a distribution more amenable to on-policy learning, enabling SFT to achieve performance rivaling and often surpassing strong on-policy baselines across diverse tasks. The key result is that by tailoring the data distribution rather than modifying the learning objective, SFT can effectively leverage privileged off-policy information for robust generalization and capability acquisition.
Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model's ability to find successful trajectories with repeated sampling. In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data. However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner. We introduce a Markov chain Monte Carlo (MCMC) sampling algorithm that progressively transforms off-policy traces to be more on-policy given a reference model for finetuning. Across tasks like scientific skill acquisition, mathematical reasoning, and open-ended expertise, our sampling algorithm enables SFT to rival prevailing posttraining techniques, often generalizing better and forgetting less than strong on-policy baselines. In addition, the resulting finetuned models exhibit strong distributional performance and are capable of learning beyond sharpening the base model distribution. At a higher level, our approach presents sampling as a model-native operator that shapes data for learnability, offering broader utility as a general-purpose primitive throughout the posttraining stack.