Search papers, labs, and topics across Lattice.
This paper introduces dynamic epistemic entropy orchestrated erasable reinforcement learning (E^3RL) to address the autoregressive curse in large language models (LLMs) during long-horizon logical reasoning. By grounding the model's local autoregressive cross-entropy in intrinsic epistemic uncertainty and utilizing segment-level adaptive thresholds, E^3RL enables the model to excise logical defects while leveraging historical key-value cache streams. Experimental results on the DeepMath-103k dataset demonstrate that E^3RL significantly enhances exploration efficiency and sample efficiency, achieving state-of-the-art performance improvements on mathematical reasoning benchmarks.
E^3RL not only overcomes the autoregressive curse but also enhances LLMs' reasoning capabilities, achieving up to 6.514% better performance than previous state-of-the-art models.
Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse. To overcome this autoregressive cascade, in which a single early mistake can compromise all subsequent reasoning steps, we propose dynamic epistemic entropy orchestrated erasable reinforcement learning ($\text{E}^3\text{RL}$). $\text{E}^3\text{RL}$ eliminates reliance on external signals by grounding the model's endogenous local autoregressive cross-entropy as an intrinsic coordinate of epistemic uncertainty. By introducing segment-level adaptive dynamic thresholds and advantage allocation, $\text{E}^3\text{RL}$ enables the model to precisely excise localized logical defects while reusing historical key-value (KV) cache streams, thereby endowing the reasoning process with a self-healing capability. We train $\text{E}^3\text{RL}$ on the DeepMath-103k dataset. Experimental results show that $\text{E}^3\text{RL}$ reshapes the exploration efficiency of long-sequence reasoning and improves sample efficiency while maintaining linear memory overhead. On mathematical reasoning benchmarks such as AIME, $\text{E}^3\text{RL}$ achieves substantial performance gains, with the 4B and 8B parameter models surpassing previous state-of-the-art (SOTA) results by 5.349\% and 6.514\%, respectively. These findings suggest that $\text{E}^3\text{RL}$ shatters the autoregressive curse in long-sequence reasoning and establishes a theoretical and systems-level foundation for the next generation of self-healing artificial general intelligence (AGI).
