Search papers, labs, and topics across Lattice.
This paper introduces Online Self-Weighted Fine-Tuning (OSW-FT), a method that enhances standard supervised fine-tuning by incorporating online, trajectory-level weighting based on the model's current performance. By estimating the model's success rate through a limited number of inference-only rollouts, OSW-FT dynamically rescales the SFT loss, allowing for more effective updates while maintaining alignment with expert trajectories. Evaluated on the Qwen3 series across various benchmarks, OSW-FT demonstrates consistent improvements over traditional SFT, particularly for small-to-medium scale models in binary-verifiable reasoning tasks.
Fine-tuning small-to-medium LLMs can be dramatically improved with just two online rollouts, reshaping efficiency in model training.
Standard supervised fine-tuning (SFT) assigns the same explicit loss weight to every expert demonstration, regardless of the model's changing competence over training queries. Reinforcement learning (RL) based methods adapt update strength using model-generated rollouts, but often require substantially more sampling and can be unstable on hard tasks. We propose \textbf{Online Self-Weighted Fine-Tuning (OSW-FT)}, a simple method that augments SFT with online, trajectory-level weighting. For each query, OSW-FT estimates the model's current success rate using a small number of inference-only rollouts and rescales the standard SFT loss accordingly. The optimization direction remains anchored to the expert trajectory, while the update magnitude adapts online. For binary-verifiable reasoning, we connect this weighting to SFT and RL at the gradient level, inspired by variance-reduction principles. The resulting estimator is unbiased for the exact OSW-FT surrogate update for any finite rollout count, and we analyze convergence with respect to the corresponding surrogate objective. Evaluated across Qwen3 series ranging from 0.6B to 4B on multiple challenging benchmarks (e.g., AIME), OSW-FT consistently improves over SFT on small-to-medium scale models. OSW-FT offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only \textbf{2 online rollouts}.
