Search papers, labs, and topics across Lattice.
This paper introduces a novel data curation method for supervised fine-tuning (SFT) of Large Language Models (LLMs) that leverages the initial reasoning tokens to identify diverse and challenging examples, circumventing the reliance on strong reasoning models. The approach demonstrates that the loss of the first 100 reasoning tokens can effectively signal problem difficulty, allowing for the selection of high-quality training data with significantly reduced costs. Experimental validation on Qwen2.5-7B and Llama3.1-8B models shows that this method not only outperforms existing baselines by up to 1.7% but also achieves a remarkable 91% increase in token efficiency.
Identifying challenging reasoning examples using just the first 100 tokens can drastically reduce data curation costs while enhancing model performance.
Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.
