Search papers, labs, and topics across Lattice.
This paper investigates the training dynamics of Reinforcement Learning with Verifiable Rewards (RLVR) in LLMs by employing vector steering to identify a low-dimensional manifold in activation space responsible for RL-induced reasoning improvements. The study reveals that this effective manifold has limited but non-infinitely compressible capacity, with its dimensionality and input-dependent expressiveness being critical constraints that vary with injection depth, and that control directions reside in the low-variance complement of the activation principal subspace. These geometric properties are found to be consistent within a task and base model, and correlate with capability transfer across tasks, leading to the development of Alpha-Stabler, a plug-and-play framework that enhances RL training stability and gains by monitoring principal subspace intrusion and orthogonalizing activation gradients.
RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.
Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains. We uncover two geometric properties. (1) Effective Manifold Capacity: the capacity needed to reproduce RL gains can be very small but is not infinitely compressible; at extremely low capacity, intervention dimensionality and input-dependent expressiveness become key constraints, and this requirement varies with injection depth. (2) Control Manifold Separation: effective control directions lie mainly in the low-variance complement of the activation principal subspace. Within a task and base model, the learned geometry stays largely consistent across training configurations, and across tasks geometric alignment correlates with capability transfer. Experiments on 5 LLMs and 6 verifiable-reward tasks support these findings. We then propose Alpha-Stabler, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement. Alpha-Stabler stabilizes training for 2,000 steps and consistently improves RL gains, offering practical insights for robust post-training. Code: https://github.com/caiyuchen-ustc/On_Policy_Vector_Training
