Search papers, labs, and topics across Lattice.
This paper reinterprets supervised fine-tuning (SFT) as a problem of target distribution design, challenging the conventional approach of maximizing the likelihood of individual tokens in a trajectory. By introducing the Q-target framework, the authors decompose SFT supervision into two key decisions: the reliance on observed tokens and the allocation of probability mass to alternatives. The proposed Target-SFT method demonstrates superior performance across ten reasoning dataset-model settings, highlighting the efficacy of a target-based approach in SFT.
Shifting the focus from token likelihood to target distribution design reveals a more effective framework for supervised fine-tuning that consistently outperforms traditional methods.
Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution Q. Building on this view, we propose Target-SFT which constructs the training objective directly from the desired target distribution. This method consistently outperforms across the ten reasoning dataset-model settings evaluated, showing the effectiveness of this target-based approach. Overall, our formulation reveals a more fundamental design principle for SFT training and opens a broader search space for SFT objectives.
