Search papers, labs, and topics across Lattice.
This paper introduces BiCo, a training-free method for transferring task vectors across different pre-trained models by aligning input-side activations and output-side gradients. The key insight is to view task vectors as accumulated bilinear interactions, enabling task-vector transfer as a dual-space alignment problem solved via orthogonal Procrustes mappings. Experiments across vision and language tasks demonstrate that BiCo outperforms existing transfer methods, closing the performance gap with direct fine-tuning.
Forget retraining: BiCo lets you transfer task-specific knowledge between different model architectures with just a single forward-backward pass.
Fine-tuning large-scale pre-trained models is a recent prevalent paradigm for adapting general representations to specialized tasks. However, when a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused because it is tied to the parameterization of the original model, requiring another costly fine-tuning. To address this inefficiency, recent work uses task vectors, defined as the parameter difference between a fine-tuned model and its base model, to transfer expertise across models. While existing methods bridge disparate models by matching activations or gradients, a significant performance gap remains relative to direct fine-tuning, suggesting that these partial correspondences are insufficient. In this work, instead of viewing a task vector merely as a parameter offset, we revisit the formation of task vectors and show that they can be derived as accumulated bilinear interactions between input-side activations and output-side gradients. Motivated by this observation, we formulate task-vector transfer as a dual-space alignment problem and propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment. BiCo estimates orthogonal Procrustes mappings in both spaces using a single forward-backward pass on a small calibration set, without any parameter update. Across extensive computer vision and natural language processing benchmarks, BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.
