Search papers, labs, and topics across Lattice.
This paper introduces AlphaWiSE, a novel weight-space interpolation method that enables continual adaptation of multimodal models without compromising cross-modal alignment. By utilizing two frozen source checkpoints and fitting a shared scalar interpolation coefficient for aligned parameter tensors, AlphaWiSE generates a single interpolated checkpoint that maintains the original model architecture and parameter count. Experimental results demonstrate that AlphaWiSE significantly outperforms existing continual-learning baselines in audio-image-text retrieval tasks, enhancing performance across various retrieval directions and metrics.
AlphaWiSE achieves superior continual adaptation in multimodal models without sacrificing cross-modal alignment, outperforming traditional methods.
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.
DeepMind