Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Vector-Quantized (AVQ) Attention, which optimizes the allocation of codebook capacity in transformer models by adaptively refining key representations based on attention importance. By starting with a small set of codewords and dynamically identifying and enhancing the most critical ones during the forward pass, AVQ Attention achieves a more efficient representation of high-attention regions while minimizing waste in low-attention areas. The implementation leverages custom Triton kernels to maintain the computational efficiency of $\mathcal{O}(MN)$ complexity while improving accuracy-efficiency trade-offs compared to traditional fixed-codebook VQ-attention methods.
AVQ Attention refines key representations dynamically, achieving better accuracy and efficiency without increasing computational complexity.
The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but applies uniform codebook capacity regardless of where attention mass concentrates: high-attention regions of key space may be coarsely approximated while low-attention regions waste representational capacity. We propose Adaptive Vector-Quantized (AVQ) Attention, which adaptively allocates codebook capacity based on attention importance. Starting from a small set of codewords, our method identifies the most important codes during the forward pass and refines them with pre-learned child codewords, achieving fine-grained quantization where it matters most while maintaining coarse quantization elsewhere. We develop an implementation using custom Triton kernels that enables the full adaptive refinement process, including importance scoring, child codeword insertion, and parent contribution replacement, to be carried out within the tiled computation paradigm of Flash Attention with minimal overhead. Our approach maintains $\mathcal{O}(MN)$ complexity while achieving improved accuracy-efficiency trade-offs compared to fixed-codebook VQ-attention.
