Search papers, labs, and topics across Lattice.
The paper introduces Pair-In, Pair-Out (PIPO), a novel approach that unifies latent compression and multi-token prediction (MTP) to accelerate LLM inference. PIPO trains a confidence head using On-Policy Distillation (OPD) to eliminate the need for a costly verifier pass in MTP, by predicting whether draft tokens should be accepted. Experiments on multiple benchmarks using Qwen3.5 models demonstrate that PIPO achieves significant speedups (up to 2.64x first-token-latency and 2.07x per-token-latency) while also improving pass@4 scores.
LLMs can be sped up by over 2x without sacrificing accuracy, by compressing the input and predicting multiple output tokens at once using a unified framework.
Long chain-of-thought reasoning has made autoregressive decoding the dominant inference cost of modern large language models. Existing methods target either the input side (latent compression) or the output side (speculative decoding and multi-token prediction, MTP), but the two lines of work have been pursued independently. Moreover, output-side methods must incur an expensive verifier pass to validate the unreliable draft tokens predicted by MTP. To address these issues, we propose \textbf{Pair-In, Pair-Out (PIPO)}, which unifies both sides by viewing a latent compressor and an MTP head as mirror-image operations: the compressor folds two input tokens into one latent representation, while the MTP head unfolds one hidden state into one additional output token. To remove the verifier cost without sacrificing reliability, PIPO trains a lightweight confidence head that decides whether draft tokens should be accepted. We observe that On-Policy Distillation (OPD) naturally matches the rejection-sampling criterion of speculative decoding, so the confidence head can be trained alongside OPD with negligible extra cost. Experiments on AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 with Qwen3.5-4B and 9B backbones show that PIPO improves pass@4 over regular decoding by up to $+7.15$ points, while delivering up to $2.64\times$ first-token-latency and $2.07\times$ per-token-latency speedups.
