Search papers, labs, and topics across Lattice.
This study critically evaluates next-chunk reasoning reinforcement learning (RL) against a simpler mixed supervised fine-tuning (SFT) approach using no-CoT data. The authors find that mixed SFT not only surpasses next-chunk reasoning RL in post-RL performance but does so with over 60 times less training compute. Their findings challenge the assumption that RL formulations inherently lead to better performance, emphasizing the importance of evaluating training strategies within the entire post-training context.
Mixed supervised fine-tuning outperforms next-chunk reasoning RL with significantly less compute, reshaping our understanding of effective training strategies for reasoning tasks.
Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.
