Search papers, labs, and topics across Lattice.
This paper introduces K-Forcing, a novel push-forward language modeling approach that enables joint next-k-token decoding, addressing the inefficiencies of traditional autoregressive (AR) models in high-load batch serving scenarios. By distilling an AR model into a conditional mapping that generates multiple tokens in a single forward pass, K-Forcing achieves significant speedups of 2.4-3.5x while maintaining compatibility with existing AR infrastructure. The method demonstrates that, despite some quality degradation, it provides a practical solution for accelerating inference in large language models, which is critical as inference costs rise in industrial applications.
K-Forcing accelerates token generation by 2.4-3.5x without abandoning the autoregressive backbone, making it a game-changer for high-load deployments.
Autoregressive (AR) language modeling is the dominant paradigm for text generation, yet its sequential token-by-token decoding makes inference memory-bound and inefficient. Existing acceleration approaches, such as speculative decoding and diffusion language models, can yield speedups under certain conditions but do not directly address high-load batch serving--the scenario most critical for industrial-scale deployment. We introduce K-Forcing, a push-forward language modeling paradigm for joint next-k-token decoding. K-Forcing distills an existing AR model into a conditional push-forward mapping--one that transforms independent uniform noise variables into a joint sample of multiple future tokens in a single forward pass. This design preserves fixed-length outputs, reuses the AR teacher backbone, and remains compatible with standard AR serving infrastructure. We train this mapping via progressive self-forcing distillation, which gradually expands the prediction window while enabling the student to closely match the sequence distribution of the AR teacher. We evaluate K-Forcing on LM1B and OpenWebText using a standard causal Transformer backbone. When aggressively configured to generate k = 4 tokens per forward pass, K-Forcing delivers approximately 2.4-3.5x speedup across different batch sizes, while incurring modest quality degradation relative to its AR teacher. As inference increasingly dominates the lifetime compute cost of modern LLMs, K-Forcing offers a promising route toward accelerating AR generation under real-world high-load deployment.
DAMO