Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FOLD, a training-free approach for speculative decoding that addresses the degradation of lightweight drafters on long contexts by treating distant tokens as shared "far memory" while preserving exact local positions. By sharpening attention within this far memory and controlling its weight, FOLD mitigates attention dilution and significantly enhances generation speed. Across diverse benchmarks up to 256K tokens, FOLD achieves substantial speedups, more than doubling autoregressive decoding performance in some cases.
Speculative decoding drafters can achieve over 2x speedup on 256K context by treating distant tokens as shared far memory with sharpened attention.
Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position. To counter attention dilution as the memory grows, we sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08$\times$ to 4.22$\times$.