Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effects of two-stage post-training methods on vision-language models, specifically focusing on the role of Stage-1 warm-start techniques like supervised fine-tuning (SFT) and on-policy distillation (OPD). The findings reveal that while Stage-1 controls the entropy regime, it does not significantly impact the final performance outcomes, as evidenced by consistent in-domain validation scores across different warm-start methods. Notably, OPD exhibits higher policy entropy and answer diversity at the start of reinforcement learning, but this advantage diminishes by the end of training, indicating that the benefits of OPD are limited and context-dependent.
Stage-1 warm-start methods may boost policy entropy but fail to translate into meaningful performance gains in downstream tasks.
Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs). We ask what Stage-1 actually controls in a small-data study using Qwen2.5-VL-7B with a same-modality 72B VLM teacher for OPD. First, the three warm-starts reach a narrow $53$--$54\%$ band on Geometry3K internal validation, consistent with the narrow range reported by recent specialized methods; this setup provides little evidence that Stage-1 changes the in-domain endpoint. Second, a matched-recipe, early-stopped SFT improves out-of-domain MathVista by $+2.1$ points, reversing the $-9.5$-point drop of an over-trained variant. The clearest difference is the \emph{entropy regime}: OPD enters RL with substantially higher policy entropy than either SFT initialization, and the separation remains visible through the available trajectories. At the in-domain initialization, OPD also has higher answer diversity and pass@16 ($+2.0$ to $+5.2$ points over SFT), although problem-level bootstrap intervals show that the smaller contrast is uncertain. The advantage is absent after RL (endpoint pass@16 values within $1.1$ points) and on MathVista (six models within $1.2$ points). Our contribution is therefore a bounded empirical characterization: Stage-1 is strongly associated with the entropy regime in this setup, but the downstream payoff is small, localized, and not evidence that OPD is a better RL warm-start.