Search papers, labs, and topics across Lattice.
This paper introduces Latent-GRPO, a novel reinforcement learning method for latent reasoning that addresses instability issues arising from the absence of intrinsic latent manifolds, exploration-optimization misalignment, and latent mixture non-closure. Latent-GRPO combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection to stabilize training. Experiments on mathematical reasoning benchmarks demonstrate that Latent-GRPO outperforms both latent initialization and explicit GRPO, achieving better performance with significantly shorter reasoning chains.
Latent reasoning, previously unstable in RL, can now outperform explicit reasoning while using 3-4x shorter chains, thanks to a new method that stabilizes latent space exploration.
Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose \textbf{Latent-GRPO}, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4$\times$ shorter reasoning chains. It also achieves stronger pass@$k$ performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.
