Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Latent Prediction (HiLP), an innovative approach that enhances long-horizon reasoning in language models by incorporating a higher-level abstract latent to mitigate error accumulation during latent-space rollouts. By addressing the limitations of existing methods like Multi-Token Prediction and Next-Latent prediction, HiLP enables more coherent belief state representations over extended sequences. Experimental results reveal that HiLP significantly improves performance on coding and multi-step reasoning tasks, showcasing its potential for more efficient speculative decoding.
Hierarchical Latent Prediction reduces error accumulation in language models, enabling coherent long-horizon reasoning and more efficient decoding.
While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.
