Search papers, labs, and topics across Lattice.
The paper introduces Slow-Fast Inference (SFI), a training-free framework to accelerate autoregressive decoding by decoupling it into frequent, low-cost "fast" steps using sparse memory and infrequent, dense-attention "slow" steps triggered near semantic boundaries. SFI leverages the observation that attention support remains stable within semantically coherent spans, allowing for efficient decoding by reusing a compact sparse memory. Experiments demonstrate that SFI achieves a $1.6\times$ to $14.4\times$ increase in decoding throughput while maintaining quality comparable to full-KV attention baselines in long-context and long-chain-of-thought settings.
Autoregressive inference gets a potential 14x speed boost without retraining, thanks to a clever trick of reusing attention weights within semantically coherent chunks.
Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$--$14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.
