Search papers, labs, and topics across Lattice.
This paper introduces StreamKL, a novel GPU primitive for efficiently computing Kullback-Leibler (KL) divergence in attention distillation without the prohibitive memory costs associated with existing methods. By employing an online formulation that processes query-key tiles through on-chip SRAM, StreamKL achieves significant speed improvements鈥攗p to 43 times faster in the forward pass and 14 times in the backward pass鈥攚hile reducing the memory footprint from quadratic to constant. The results demonstrate that StreamKL enables long-context attention distillation on a single GPU, which is crucial for scaling models in various applications like model compression and continual learning.
StreamKL slashes memory usage from quadratic to constant, enabling efficient long-context attention distillation on a single GPU.
Attention distillation, which trains one attention distribution to match another by minimizing their Kullback-Leibler (KL) divergence, is widely used in knowledge distillation, model compression, continual learning, and sparse-attention LLM training. However, existing approaches materialize both attention distributions before computing the KL reduction, incurring $O(N_QN_K)$ memory and IO costs that become prohibitive at long context lengths. We present StreamKL, the first fused GPU primitive for attention KL divergence that eliminates this quadratic materialization. StreamKL derives a novel online formulation for the coupled two-distribution KL reduction, enabling a single one-pass forward kernel that streams query-key tiles through on-chip SRAM. For the backward pass, StreamKL recomputes attention probabilities tile-by-tile, avoiding storage of quadratic intermediates. We further design and implement efficient GPU kernels with dedicated optimizations. Experiments show StreamKL delivers up to $43\times$ and $14\times$ speedups over baseline methods in the forward and backward passes, respectively. Most importantly, StreamKL reduces the extra HBM footprint of attention distillation from $O(N_QN_K)$ to $O(1)$, enabling long-context distillation on a single GPU.
