Search papers, labs, and topics across Lattice.
This paper introduces a multi-reward Reinforcement Learning from Internal Feedback (RLIF) framework to improve the reasoning abilities of LLMs without external supervision. The framework decomposes the training signal into an answer-level reward (cluster voting) and a completion-level reward (token-wise self-certainty), using GDPO-based normalization to handle reward-scale imbalance. KL-Cov regularization is further introduced to prevent entropy collapse by targeting low-entropy token distributions. The method demonstrates improved stability and performance on mathematical reasoning and code-generation benchmarks compared to single-reward RLIF, approaching the performance of supervised RLVR methods.
Forget reward hacking and entropy collapse: multi-reward RLIF, combining answer-level and completion-level signals, unlocks stable and robust LLM reasoning without human supervision.
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.
