Search papers, labs, and topics across Lattice.
The paper introduces a training-free model adaptation technique that transplants internal modules from a source language model to a target model based on activation analysis. They identify task-specific modules by analyzing activation changes during inference and then transplant these modules to improve the target model's performance on the same task. Experiments demonstrate that this transplantation method can significantly improve the performance of underperforming models, sometimes exceeding the original baseline and achieving substantial gap recovery, without requiring any training or fine-tuning.
Forget fine-tuning: surgically transplanting activation-selected modules between language models can double performance and fully recover capability gaps, all without training.
Despite the continuous research and evolution of language models, they sometimes underperform previous versions. Existing approaches to overcome these challenges are resource-intensive, highlighting the need for alternatives that enable immediate action. We assume that each language model has a local module inside that is suitable for a specific function. First, this work identifies a set of modules showing consistent and local activation changes under an inference workload through activation-based analysis. Subsequently, we transplant an internal module that is properly activated for a specific task into the target model, leading to immediate and measurable functional changes without additional training or fine-tuning. To experimentally demonstrate the effectiveness of the transplant technique, we quantify the relationship between transplant strength and performance improvement under different conditions for two language models. In the cross-generation setting, we find that transplanting activation-selected modules can substantially improve the underperforming model, reaching up to twice the target baseline and achieving gap-based recovery above 100%. Moreover, in transplant experiments between a base model and its instruction-tuned counterpart, transplantation improves the underperforming model toward the stronger baseline, yielding up to about 2.33 times the target baseline with gap-based recovery reaching up to 100% in the best case. These results show that meaningful capacity transfer can be realized through the implantation of highly localized modules implied by language models. Overall, this work provides empirical evidence for task-localized modularity in language models and presents a new research area: model transplantation.
