Search papers, labs, and topics across Lattice.
This study investigates the performance of Small Language Models (SLMs) fine-tuned with Reinforcement Learning with Verifiable Rewards (RLVR) in low data and compute environments. By analyzing three novel datasets focused on number counting, graph reasoning, and spatial reasoning, the authors reveal that models trained on simpler tasks can effectively generalize to more complex tasks, particularly when utilizing mixed complexity datasets. The results indicate that procedural datasets can enhance sample efficiency by up to 5x in low data scenarios, highlighting a significant advancement in model training methodologies under resource constraints.
Training on mixed complexity datasets can yield up to 5x sample efficiency in low data regimes, challenging conventional wisdom about data quantity in LLM fine-tuning.
Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers in the case of Reinforcement Learning with Verifiable Rewards (RLVR). While previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for RLVR, these results lack applicability in many real-world settings where annotated data and accessible compute may be scarce. In this work, we present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes. Across three novel datasets covering number counting problems, graph reasoning, and spatial reasoning, we characterize how model performance scales with dataset size, diversity, and complexity. We demonstrate that (1) procedural datasets allow for fine-grained evaluation and training dataset development with controllable properties (size, diversity, and complexity), (2) under RLVR, models trained on lower complexity tasks can generalize to higher complexity tasks, and (3) training on mixed complexity datasets is associated with the greatest benefits in low data regimes, providing up to 5x sample efficiency versus training on easy tasks. These findings inspire future work on the development of data scaling laws for RLVR and the use of procedural data generators to further understand effective data development for efficient LLM fine-tuning.
