Search papers, labs, and topics across Lattice.
This paper introduces L1 augmented attention, a modification to the standard dot product attention mechanism in Transformer models that incorporates a learned L1 distance metric to enhance vector similarity assessments. By projecting queries and keys into low-dimensional subspaces, the method effectively captures both directional alignment and coordinate deviations, leading to improved performance. Evaluations on the WikiText 2 dataset show that this approach can achieve up to a 14.5% reduction in perplexity compared to traditional transformers, highlighting its potential for enhancing language model accuracy and efficiency.
L1 augmented attention achieves a remarkable 14.5% reduction in perplexity by integrating L1 geometry into Transformer models, challenging the dominance of traditional dot product methods.
Scaled dot product attention conflates directional alignment and vector magnitude, limiting its effectiveness as a similarity metric in Transformer models. We introduce L1 augmented attention, a simple and computationally parallelizable modification that subtracts a learned, head specific L1 distance between queries and keys from the dot product score. This hybrid similarity captures complementary geometric information. Dot product rewards directional alignment, while L1 penalizes coordinate deviations. To reduce the cost of L1 computation, we project queries and keys into low dimensional subspaces whose parameters specialize to preserve informative L1 structure. Evaluated on WikiText 2 using a compact transformer, L1 augmented attention achieves up to a 14.5% reduction in perplexity over the original transformer baseline and outperforms an RBF L2 kernel. Analysis of norm variance and learned L1 weights reveals distinct geometric roles across layers and strong head level specialization. These results demonstrate that enriching attention with L1 geometry provides a principled and effective improvement to similarity computation in modern language models, with practical benefits for both accuracy and parallel efficiency.
