Search papers, labs, and topics across Lattice.
This paper introduces DIAG, a framework that enhances the efficiency of iterative preference optimization for aligning Large Language Models (LLMs) in mathematical reasoning tasks by adaptively reshaping the practice distribution. By diagnosing valid preference-pair yield and generating targeted practice based on a student's failure traces, DIAG effectively focuses training on the student's current competence boundary. Experimental results indicate that DIAG increases the yield of informative preference pairs and improves reasoning performance without exceeding the training budget, showcasing its potential for more effective mathematical preference distillation.
DIAG reshapes practice distribution to maximize informative supervision, leading to significantly improved reasoning performance in LLMs.
Iterative preference optimization is essential for aligning Large Language Models on mathematical reasoning tasks, yet its efficiency is often throttled by signal scarcity: as the model improves, static problem sets become increasingly mismatched to the model's evolving competence, producing rollouts that are either too easy or too hard and therefore non-informative, which leads to a scarcity of valid preference pairs. We propose DIAG, a Diagnostic Iterative Alignment and Generation framework that adaptively reshapes the practice distribution to increase informative supervision and focus training near the student's current competence boundary. DIAG consists of two phases: (1) diagnosing valid preference-pair yield to calibrate the exploration-exploitation trade-off and allocate topic quotas via an Empirical Bayes shrinkage estimator, thereby prioritizing high-yield concepts; and (2) generating targeted practice, where a teacher synthesizes variants from the student's failure traces. We further provide a theoretical view interpreting DIAG as a teacher-mediated approximation to KL-regularized reweighting of the practice distribution toward the student's competence boundary, where valid preference-pair yield is maximized. Experiments show that DIAG boosts yield across iterations and delivers stronger reasoning performance under an iso-effective training budget, demonstrating that it can distill more informative preference supervision for mathematical reasoning.
Tsinghua AI