Search papers, labs, and topics across Lattice.
This paper addresses the instability in online reinforcement learning for aligning large language models (LLMs) caused by a phenomenon termed directional inconsistency, where a few high-reward rollouts create conflicting preference directions within a batch. The authors introduce geoalign, a lightweight plug-in that curates rollouts by forming preference pairs, learning an online projector for reward-ordered displacement directions, and rectifying directionally inconsistent rollouts with stable alternatives. The implementation of geoalign significantly enhances performance and reduces training oscillation compared to existing methods like PF-PPO and PAR, indicating that leveraging latent directional consensus can improve the reliability of online LLM reinforcement learning.
Directional inconsistency can destabilize LLM training, but geoalign curates rollouts to improve performance and stability, outperforming several established methods.
Online reinforcement learning is widely used to align large language models (LLMs) with reward signals, yet training can be unstable under noisy or misspecified rewards. We identify a failure mode we call directional inconsistency: within a batch, a small set of high-reward rollouts induces representation-space preference directions that sharply disagree with the batch majority, resulting in high-variance and destabilizing updates. We propose geoalign, a lightweight plug-in for rollout curation in iterative policy optimization. Geoalign (i) forms within-prompt preference pairs, (ii) learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and (iii) detects directionally inconsistent rollouts via their angular deviation from a batch consensus prototype and rectifies them with within-prompt stable alternatives. Geoalign is forward-pass only and adds negligible overhead. Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation, outperforming PF-PPO, PAR, PODS, and Seed-GRPO. These results suggest latent directional consensus as an effective reliability signal for online LLM RL.
