Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces RL2ML, a family of finite-rollout surrogate objectives for training language models from binary feedback, bridging reinforcement learning and maximum likelihood estimation. RL2ML provides a closed-form, exactly unbiased gradient estimator and allows for continuous interpolation between RL and beyond-MLE objectives while maintaining estimator-objective alignment. Through group-level update scale analysis, calibrated metric-gain analysis, and variance decomposition, the authors demonstrate that the optimal surrogate objective depends on the evaluation metric, local sensitivity, and estimator variance, leading to a one-dimensional optimization problem.
Forget choosing between RL and MLE objectives for language model training from binary feedback – RL2ML lets you smoothly interpolate between them and beyond, optimizing for metric gain and estimator variance.
Correctness-based Reinforcement Learning with Verifiable Rewards (RLVR) trains language models from binary feedback on sampled outputs, but the objective optimized in expectation and the stochastic update geometry induced by finite rollout groups are often conflated. This paper develops RL2ML, a family of finite-rollout surrogate objectives with a closed-form, exactly unbiased gradient estimator. The family continuously connects standard reinforcement learning, maximum-likelihood-like training, and beyond-maximum-likelihood objectives while preserving estimator-objective alignment under a fixed rollout budget. We introduce the group-level update scale to characterize how a rollout group is reweighted after its empirical success count is observed, revealing a subcritical-supercritical update-scale transition that is hidden by population-level objective notation alone. Building on this distinction, calibrated metric-gain analysis and exact variance decomposition show that the best choice of surrogate objective is determined neither by proximity to maximum likelihood nor by the population-level weight alone. Instead, it depends jointly on the evaluation metric, local sensitivity, and estimator variance. The remaining degree of freedom in the surrogate objective family can therefore be formulated as a one-dimensional optimization problem rather than treated as an unconstrained hyperparameter.