Search papers, labs, and topics across Lattice.
This paper introduces the concept of free pause tokens, which enhance next-token prediction in language models by utilizing a parallel prediction stream over a weight-shared backbone, rather than adding extra tokens to the sequence. The implementation shows a 2-3 centinat improvement in prediction accuracy on a 1B parameter model, while incurring minimal additional costs in terms of context length, KV cache, and latency during inference. Notably, the training overhead is only slightly increased, making this approach efficient and effective for improving transformer performance without significant resource penalties.
Free pause tokens boost language model performance without increasing context length or latency, achieving significant gains with minimal training overhead.
A free pause token gives a language model extra compute to form each next-token prediction (as a pause, or thinking, token does) but carries that compute in a parallel prediction stream over a weight-shared backbone rather than as an extra token in the sequence. It improves next-token prediction by 2-3 centinats in practice on a 1B parameter model. Because the pause rides an existing position instead of adding one, it is free to use: at inference it adds no context length, no KV cache, and essentially no latency with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The only primary cost is in training, where additional training compute versus an optimized pretraining pipeline is reduced to as low as x1.14 while preserving most of the benefits. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers.
Microsoft Research