Search papers, labs, and topics across Lattice.
This paper introduces GCPO (Geometrically Constrained Policy Optimization), a method that addresses the training instabilities and performance degradation associated with on-policy rollout methods like GRPO in large language models. By employing Principal-Subspace Overlap to analyze the geometry of individual rollout updates, the authors demonstrate that constraining updates to complementary subspaces can stabilize performance and prevent excursions that lead to degradation. Experimental results show that GCPO outperforms GRPO and its recent variants across various tasks, achieving significant improvements while maintaining general capabilities and stabilizing policy entropy.
Constraining rollout updates to complementary subspaces can enhance performance and stability in large language models, achieving up to 27.69 points improvement over existing methods.
On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.
