Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Soft Latent Thinking, a novel approach that replaces the traditional large vocabulary head in large language models with a lightweight projector, allowing reasoning to occur in continuous embedding space rather than through discrete tokens. By conducting experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B, the authors demonstrate that this method not only enhances performance across various pass@k metrics but also significantly reduces computational costs during chain-of-thought reasoning. The results reveal that effective reasoning can be achieved in continuous space, achieving the highest pass@32 among existing soft-thinking methods, thus challenging the conventional reliance on tokenized outputs.
Continuous reasoning in embedding space can outperform traditional token-based methods while slashing computational costs.
Large language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all reasoning to be expressed in discrete tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective reasoning can be carried out in continuous space without discrete token generation.