Search papers, labs, and topics across Lattice.
This paper introduces Surrogate Latent Policy Optimization (SLPO), a novel approach that integrates outcome-reward reinforcement learning with latent reasoning to enhance test-time scaling in autoregressive models. By employing an empirical surrogate policy density for trajectory-level credit assignment and a correctness-supervised stopping mechanism, SLPO enables latent reasoners to allocate computation more effectively, particularly for challenging tasks. The results demonstrate that SLPO significantly improves performance metrics like Pass@$k$ while allowing for longer computation on more complex instances, surpassing traditional explicit Chain-of-Thought methods.
Latent reasoning can now leverage outcome-reward reinforcement learning, achieving better performance with less computational overhead than traditional methods.
Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@$k$ under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
Tsinghua AI