Search papers, labs, and topics across Lattice.
The paper introduces Delta Attention Selective Halting (DASH), a training-free method that reduces prefilling costs in LLMs by selectively halting computation for tokens that have converged to "semantic fixing points" based on layer-wise attention update dynamics. DASH preserves hardware efficiency by remaining compatible with kernels like FlashAttention, unlike prior token pruning heuristics. Experiments across language and vision benchmarks demonstrate significant prefill speedups without compromising model accuracy.
LLMs waste compute on tokens that have already "figured it out" – DASH selectively skips these tokens during prefill, speeding things up without retraining or sacrificing accuracy.
Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.
CMU ML