Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
To eliminate expensive from-scratch pre-training for linear Vision Transformers, this work investigates how to effectively transfer pre-trained weights across the architectural boundary from standard Softmax ViTs. The authors find that while attention weights are strictly operator-specific and fail under naive transfer, MLP representations are operator-agnostic and can be directly copied, while the attention routing behavior is recoverable via distillation. Combining direct MLP weight inheritance with targeted attention distillation allows linear ViTs to close the efficiency-accuracy trade-off, consistently matching or outperforming their Softmax counterparts across diverse model scales and benchmarks.
Pre-trained Softmax ViTs can directly bootstrap linear-attention models without expensive pre-training from scratch: simply copy the operator-agnostic MLP weights and distill the attention routing behavior.
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.