Search papers, labs, and topics across Lattice.
The paper introduces Spectral Editing Key Amplification (SEKA), a training-free attention steering method that modifies key embeddings using spectral decomposition to amplify attention scores for specific tokens without requiring explicit storage of the full attention matrix. This addresses the incompatibility of existing attention steering methods with memory-efficient attention mechanisms like FlashAttention. Experiments demonstrate that SEKA and its adaptive variant, AdaSEKA, outperform strong baselines on steering benchmarks with lower latency and memory overhead, making them compatible with optimized attention implementations.
Steer LLM attention without the memory bottleneck: SEKA unlocks prompt highlighting and other control capabilities even with FlashAttention.
Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
