Search papers, labs, and topics across Lattice.
This paper addresses the challenge of training large language models for efficient reasoning by introducing ACOER (Adaptive Correct-Only Efficiency Reward), which mitigates issues of reward collapse associated with traditional length-penalizing strategies in Group Relative Policy Optimization (GRPO). By isolating brevity bonuses to correct completions and implementing dynamic budget normalization, ACOER enhances model accuracy while significantly reducing token generation. The results show that ACOER not only improves reasoning performance across various benchmarks but also establishes a more stable framework for efficiency-aware optimization in large models.
ACOER reduces token generation by over 60% while boosting accuracy, solving the reward collapse problem that plagues traditional efficiency training methods.
Training large language models to reason efficiently is a critical challenge. While integrating length-penalizing rewards into Group Relative Policy Optimization (GRPO) aims to reduce verbosity, it frequently triggers reward collapse, severely degrading reasoning capabilities. Through a systematic evaluation of various reward configurations, we identify the root mechanism: GRPO's group normalization creates divergent advantages when incorrect answers receive continuous length penalties. Consequently, methods penalizing the length of incorrect answers are structurally prone to collapse under sustained optimization. Furthermore, restricting penalties exclusively to correct answers avoids this primary failure, but leaves the model susceptible to a stochastic collapse driven by response over-compression. To robustly prevent both failure modes, we propose ACOER (Adaptive Correct-Only Efficiency Reward). ACOER eliminates the structural penalty loop by isolating brevity bonuses to correct completions and prevents stochastic compression via dynamic budget normalization and control-loop penalty adjustments. Evaluated across diverse mathematical reasoning benchmarks, ACOER improves overall accuracy compared to the base model while reducing token generation by over 60%, establishing a fundamentally stable approach for efficiency-aware optimization.
