Search papers, labs, and topics across Lattice.
This paper introduces a scalable framework for merging billion-parameter pretrained transformers using linear mode connectivity (LMC) by employing parameterized weight transformations and a dual learning procedure. The method allows for joint optimization of two models to create a shared linear interpolation path, significantly reducing loss barriers during merging. Empirical results demonstrate near-zero loss barriers for language models on WikiText and robust performance for vision models like ViT-L on ImageNet, marking a significant advancement in model merging techniques at scale.
Achieving near-zero loss barriers in model merging could revolutionize how we connect and utilize billion-parameter transformers.
Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the interpolation path from only one model endpoint, limiting their scalability and effectiveness for large pretrained transformers. We propose a novel and scalable framework for enabling LMC-based model merging to {\em billion-parameter pretrained transformers}. Our method applies properly parameterized functionality-preserving weight transformations to align functionally equivalent solutions, and introduces a dual learning procedure in which both models jointly learn their corresponding transformations toward a shared linear interpolation path. This bidirectional optimization substantially reduces interpolation barriers and enables more reliable merging across large-scale architectures. Empirically, we show that our approach achieves near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale. In the vision domain, ViT-L maintains above 69\% ImageNet top-1 accuracy throughout the interpolation path, while modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that properly resolving parameter symmetries enables large pretrained Transformers to be connected and merged through simple linear paths with substantially improved interpolation performance. Code: https://github.com/VILA-Lab/Dual-Learned-Matching .
