Search papers, labs, and topics across Lattice.
This paper introduces the Hybrid Gated Attention (HyGA) framework, which integrates three distinct gating strategies to enhance attention mechanisms by capturing intra-head and cross-head interactions. By employing low-rank matrix decomposition and learnable attention sinks, HyGA not only improves training efficiency and stability but also significantly boosts representational capacity. Experimental results demonstrate that HyGA outperforms traditional gated attention across various benchmarks, achieving superior performance at different computational costs.
HyGA achieves unprecedented improvements in attention mechanisms by effectively integrating multiple gating strategies, leading to enhanced representational capacity and training efficiency.
Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its effectiveness-efficiency Pareto frontier, we propose a Hybrid Gated Attention (HyGA) framework that contains three types of gating strategies. Specifically, these gates leverage diverse information from multiple stages of attention, and collaboratively build element-wise/head-wise gating from multiple perspectives, capturing intra-head and cross-head information interactions. Through our hybrid gating components, HyGA could provide multi-source modulation signals, enabling more comprehensive control over information flow and improving the representational capacity of attention. We also introduce low-rank matrix decomposition and learnable attention sink to further enhance training efficiency and stability. In experiments, we evaluate HyGA on widely-used benchmarks based on different backbones. The experimental results show that our HyGA comprehensively improves both training loss and various downstream performances compared with Gated attention. HyGA has also been verified to achieve the best performance at different computation costs, with comprehensive model analyses for better understanding. The proposed HyGA sheds light on a more effective, efficient, and stable attention mechanism.
