Search papers, labs, and topics across Lattice.
This paper introduces TARPO, a novel reinforcement learning framework that combines discrete token generation with continuous latent reasoning to enhance policy exploration in large language models. By employing a lightweight action head router that dynamically switches between reasoning modes, TARPO achieves superior performance compared to existing methods across various benchmarks. The results indicate that TARPO not only improves reasoning capabilities but also learns adaptive token-wise switching behaviors, ensuring stable training dynamics.
TARPO outperforms traditional reasoning methods by seamlessly integrating discrete and continuous approaches, revolutionizing policy exploration in LLMs.
Latent reasoning has emerged as a promising alternative to discrete Chain-of-Thought (CoT) in large language models (LLMs), enabling more expressive reasoning by operating over continuous representations. However, the inherently deterministic nature of continuous representations limits policy exploration in reinforcement learning (RL). To address this, we propose TARPO (Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization), a pure RL framework that adaptively switches between discrete token generation and continuous latent reasoning at each step. TARPO introduces a lightweight action head router that observes the current hidden state and samples a routing decision from a binary mode-selection space, preserving the stochasticity of discrete token sampling from the vocabulary. The LLM backbone and router are jointly optimized end-to-end with a shared group-relative advantage signal. Extensive experiments across Qwen2.5 (from 1.5B to 7B) and Llama-3.1-8B backbones demonstrate that TARPO consistently outperforms existing explicit and latent reasoning RL baselines across diverse benchmarks. Further analysis shows that TARPO learns adaptive token-wise switching behaviors while maintaining stable training dynamics. Our code is available at https://github.com/NKU-LITI/TARPO-master.
