Search papers, labs, and topics across Lattice.
QFlash introduces an end-to-end integer-only FlashAttention, enabling full quantization of the attention mechanism in Vision Transformers. It overcomes challenges like scale explosion and inefficient integer exponential operations by performing softmax entirely in the integer domain within a single Triton kernel. Experiments on ViT, DeiT, and Swin models show QFlash achieves up to 6.73x speedup over I-ViT and 8.69x speedup on Swin, with reduced energy consumption and competitive accuracy under per-tensor quantization.
Integer-only attention is now a viable alternative to floating-point, delivering up to 8.69x speedups and 18.8% energy reduction on Vision Transformers.
FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison. To address these challenges, we propose \textit{QFlash}, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel. On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73$\times$ speedup over I-ViT and up to 8.69$\times$ speedup on Swin, while reducing energy consumption by 18.8\% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization. Our code is publicly available at https://github.com/EfficientCompLab/qflash.
