Search papers, labs, and topics across Lattice.
This paper introduces LoopMTP, a novel looped transformer architecture that enhances reasoning capabilities by integrating multi-token prediction (MTP) to guide intermediate representations across iterations. By reusing a single stack of layers through multiple iterations, LoopMTP achieves the effective depth of larger models without increasing parameter count, addressing issues of latent overthinking and undifferentiated computation. The results demonstrate an improvement in average accuracy of up to 8.1% relative to non-looped baselines, while maintaining training stability across 15 loops.
LoopMTP boosts reasoning accuracy by up to 8.1% by effectively guiding looped transformer iterations with multi-token prediction.
Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across $T$ iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops $T$ times can anticipate $T$ future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1\% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.
