Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces two modifications to the transformer attention block: a non-linear pre-projection MLP before Q/K/V projections to construct richer, position-agnostic features, and a content skip connection that allows content information to bypass the attention mechanism. The pre-projection MLP constructs richer features before positional information is applied, while the content skip allows content information to bypass position-aware attention. Experiments on Pythia-160M and 410M show that the combined approach achieves significant improvements in LAMBADA accuracy (+40.6%) and perplexity (-39%) at the 160M scale, with no K/V cache overhead.
Skipping the attention mechanism can boost transformer performance by 40% without adding K/V cache overhead, especially in later layers.
We propose two complementary modifications to transformer attention blocks. First, a non-linear pre-projection MLP is inserted between layer norm and Q/K/V projections, constructing richer features in a position-agnostic manner before any positional encoding is applied. Second, a content skip connection routes the pre-projection's features around the attention mechanism, allowing content information to bypass position-aware attention where beneficial. In frozen-probe experiments on Pythia-160M and 410M, the combined approach achieves the strongest results across methods: +40.6% LAMBADA accuracy and -39% perplexity at 160M scale. Learned skip connection weights reveal a consistent pattern across model sizes: later transformer layers activate the content bypass more strongly than earlier layers, suggesting that deeper layers benefit from content information that does not pass through positional attention. All modifications add no K/V cache overhead.