Search papers, labs, and topics across Lattice.
The paper introduces Historical Attention Vision Transformer (HAViT), a method that propagates and integrates historical attention matrices across ViT encoder layers to refine inter-layer information flow. By storing and blending attention matrices, HAViT enables progressive refinement of attention patterns, leading to enhanced feature acquisition. Experiments on CIFAR-100 and TinyImageNet demonstrate consistent accuracy improvements of over 1% across various ViT architectures.
Injecting "historical attention" into vision transformers boosts accuracy by over 1% with minimal architectural changes, suggesting that current ViTs underutilize information learned in earlier layers.
Vision Transformers have excelled in computer vision but their attention mechanisms operate independently across layers, limiting information flow and feature learning. We propose an effective cross-layer attention propagation method that preserves and integrates historical attention matrices across encoder layers, offering a principled refinement of inter-layer information flow in Vision Transformers. This approach enables progressive refinement of attention patterns throughout the transformer hierarchy, enhancing feature acquisition and optimization dynamics. The method requires minimal architectural changes, adding only attention matrix storage and blending operations. Comprehensive experiments on CIFAR-100 and TinyImageNet demonstrate consistent accuracy improvements, with ViT performance increasing from 75.74% to 77.07% on CIFAR-100 (+1.33%) and from 57.82% to 59.07% on TinyImageNet (+1.25%). Cross-architecture validation shows similar gains across transformer variants, with CaiT showing 1.01% enhancement. Systematic analysis identifies the blending hyperparameter of historical attention (alpha = 0.45) as optimal across all configurations, providing the ideal balance between current and historical attention information. Random initialization consistently outperforms zero initialization, indicating that diverse initial attention patterns accelerate convergence and improve final performance. Our code is publicly available at https://github.com/banik-s/HAViT.
