Search papers, labs, and topics across Lattice.
This study explores the enhancement of reasoning language models (RLMs) through a novel approach that combines instruction tuning with model merging. By leveraging vast amounts of unused supervised fine-tuning data, the authors demonstrate that RLM performance can be significantly improved in both verifiable and challenging domains, such as coding and text summarization. The results indicate that this method not only boosts performance but also maintains the reasoning capabilities of the models across other areas, all at a minimal cost of under USD $3.
Unlocking the potential of unused fine-tuning data can enhance reasoning models' performance in complex domains without sacrificing their core capabilities.
Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.
