Search papers, labs, and topics across Lattice.
This paper addresses the issue of Positive-Credit Contamination in reinforcement learning for large language models, where low-probability tail tokens receive the same positive credit as plausible tokens, leading to flawed reasoning. The authors introduce Tail-Aware Credit Calibration (TACO), which calibrates credit assignment by computing a tail-risk score to differentiate between useful rare patterns and unreliable tokens. Experimental results demonstrate that TACO significantly enhances training stability and performance across multiple LLMs and benchmarks, outperforming traditional GRPO-style methods.
TACO effectively mitigates the reinforcement of erroneous reasoning in LLMs by distinguishing between useful and unreliable tokens, leading to improved training stability and performance.
Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: https://github.com/xiuyilou/TACO.
