Search papers, labs, and topics across Lattice.
This survey paper provides a structured overview of model merging techniques for large language models (LLMs), categorizing methods using the FUSE taxonomy (Foundations, Unification Strategies, Scenarios, Ecosystem). It covers theoretical underpinnings like loss landscape geometry and linear mode connectivity, and reviews various merging algorithms including weight averaging, task vector arithmetic, and mixture-of-experts. The paper also examines applications in multi-task learning, safety alignment, and domain specialization, while highlighting open challenges like scalability and standardization.
Forget ensembling or retraining: model merging lets you Frankenstein LLMs for specialized skills at minimal cost.
Model merging has emerged as a transformative paradigm for combining the capabilities of multiple neural networks into a single unified model without additional training. With the rapid proliferation of fine-tuned large language models~(LLMs), merging techniques offer a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey presents a comprehensive and structured examination of model merging in the LLM era through the \textbf{FUSE} taxonomy, a four-dimensional framework organized along \textbf{F}oundations, \textbf{U}nification Strategies, \textbf{S}cenarios, and \textbf{E}cosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry, mode connectivity, and the linear mode connectivity hypothesis. We then systematically review the algorithmic landscape, spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization approaches. For each method family, we analyze the core formulation, highlight representative works, and discuss practical trade-offs. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, multilingual transfer, and federated learning. Finally, we survey the supporting ecosystem of open-source tools, community platforms, and evaluation benchmarks, and identify key open challenges including theoretical gaps, scalability barriers, and standardization needs. This survey aims to equip researchers and practitioners with a structured foundation for advancing model merging.
