Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Control Reinforcement Learning (ACRL), a method designed to mitigate the instability in reinforcement learning training for large language models caused by discrepancies between training and inference processes. By adaptively managing the training-inference discrepancy, ACRL not only stabilizes the training but also enhances policy exploration through increased entropy. Experimental results demonstrate that ACRL effectively maintains this discrepancy within acceptable limits while matching and even surpassing the accuracy of existing benchmarks like BF16 and importance sampling fixes.
ACRL stabilizes RL training while boosting exploration, outperforming traditional methods even under low-precision inference conditions.
Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.