Search papers, labs, and topics across Lattice.
This paper introduces a novel model merging technique that formulates the merging process as computing a weighted Karcher mean on the Fisher-Rao manifold, aiming to minimize KL divergence between predictive distributions. They derive a fixed-point algorithm using a spherical proxy to address limitations of parameter-space heuristics in existing weight-space merging methods. Experiments demonstrate that their method outperforms existing baselines in stability and accuracy, especially when merging a large number of heterogeneous models, while also mitigating representation collapse.
By merging models on the Fisher-Rao manifold, this work achieves stable and accurate LLM merging even with many heterogeneous models, overcoming the representation collapse issues plaguing simpler weight averaging techniques.
Weight-space merging aims to combine multiple fine-tuned LLMs into a single model without retraining, yet most existing approaches remain fundamentally parameter-space heuristics. This creates three practical limitations. First, linear averaging, task vectors, and related rules operate on Euclidean coordinates, even though the desired goal is to merge functionality, i.e., predictive behaviors across tasks. Second, when the source checkpoints are farther apart or more heterogeneous, Euclidean blends often trigger representation collapse, manifested as activation variance shrinkage and effective-rank degradation, which sharply degrades accuracy. Third, many geometry-inspired methods are most natural for two-model interpolation and do not extend cleanly to merging N>2 experts with a principled objective. We address these issues by formulating model merging as computing a weighted Karcher mean on the Fisher--Rao manifold, which is locally equivalent to minimizing a KL-based function distance between predictive distributions. We derive a practical fixed-point algorithm using a lightweight spherical proxy that preserves norms and generalizes directly to multi-expert merging. Across various benchmarks and collapse diagnostics, our method remains stable as the number and heterogeneity of merged models increase, consistently outperforming prior baselines.
