Search papers, labs, and topics across Lattice.
This paper introduces Dynamic Important Example Mining (DIEM), a novel framework for reinforcement fine-tuning (RFT) that dynamically adapts the selection of training samples based on their marginal contribution to policy improvement. By integrating a gradient-alignment importance estimator and a constrained batch reweighting scheme, DIEM optimizes data utilization throughout the training process, addressing the limitations of static sample selection methods. Experimental results demonstrate that DIEM significantly outperforms both static and dynamic baselines across various reasoning benchmarks, highlighting its effectiveness in enhancing model performance.
Static sample selection in reinforcement fine-tuning is a recipe for suboptimal updates鈥擠IEM adapts dynamically, leading to superior performance on reasoning tasks.
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.
Stanford HAI