Search papers, labs, and topics across Lattice.
The paper introduces a function-preserving expansion method to mitigate catastrophic forgetting during fine-tuning by replicating pre-trained parameters within transformer submodules and applying a scaling correction. This ensures the expanded model is functionally identical to the original at initialization, promoting stable training and leveraging existing knowledge. Experiments show that the method matches full fine-tuning performance on downstream tasks without compromising original capabilities, and selective layer expansion achieves comparable results at a reduced computational cost.
Forget catastrophic forgetting: this function-preserving expansion method lets you fine-tune without sacrificing pre-trained knowledge, matching full fine-tuning performance at a fraction of the cost.
Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromise performance on the new task or struggle to balance training stability with efficient reuse of pre-trained knowledge. We introduce a novel function-preserving expansion method that resolves this dilemma. Our technique expands model capacity by replicating pre-trained parameters within transformer submodules and applying a scaling correction that guarantees the expanded model is mathematically identical to the original at initialization, enabling stable training while exploiting existing knowledge. Empirically, our method eliminates the trade-off between plasticity and stability, matching the performance of full fine-tuning on downstream tasks without any degradation of the model's original capabilities. Furthermore, we demonstrate the modularity of our approach, showing that by selectively expanding a small subset of layers we can achieve the same performance as full fine-tuning at a fraction of the computational cost.
Google Research