Search papers, labs, and topics across Lattice.
This paper introduces ConSA, a framework that optimally allocates full attention (FA) and sliding-window attention (SWA) in hybrid architectures by learning binary masks under a specified sparsity target. Utilizing L0 regularization and an augmented Lagrangian constraint, ConSA allows for a more nuanced allocation strategy that significantly outperforms traditional rule-based methods. The findings reveal that learned allocations favor placing SWA in lower layers while concentrating FA in contiguous middle-layer blocks, highlighting a spectrum of intrinsic attention behaviors across different model scales and sparsity levels.
Learned attention allocation patterns reveal that SWA is best positioned in lower layers, challenging conventional wisdom on attention distribution in LLMs.
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely on hand-crafted rules or simple post-hoc heuristics for FA/SWA allocation and offer limited analysis of the attention behaviors underlying these designs. We propose Controllable Sparsity in Hybrid Attention (ConSA), a framework that learns optimal FA/SWA assignment under a user-specified sparsity target. ConSA employs L0 regularization to learn binary masks selecting between FA and SWA for each attention unit, while an augmented Lagrangian constraint enforces the target sparsity at either layer or KV-head granularity. We evaluate ConSA on two LLMs at the 0.6B and 1.7B scales. Learned allocations consistently outperform rule-based baselines, with KV-head-wise allocation yielding clear gains over layer-wise allocation. The learned patterns place SWA in the bottom layers and concentrate FA into contiguous middle-layer blocks, diverging from evenly interleaved patterns in rule-based methods. This structure persists across model scales, sparsity levels, and allocation granularities, revealing a fine-grained spectrum of intrinsic attention behaviors that underlies the learned allocation.
