Search papers, labs, and topics across Lattice.
The paper introduces a dataless regularization method for Task Arithmetic to mitigate cross-task interference and representation drift when combining multiple task vectors in foundation models. They frame representation drift regularization as a curvature matrix approximation problem, specifically using Kronecker-Factored Approximate Curvature (K-FAC). The proposed method achieves state-of-the-art results in task addition and negation while maintaining constant complexity in the number of tasks and robustness to task vector rescaling.
Achieve state-of-the-art task arithmetic without any data by using Kronecker-Factored Approximate Curvature to disentangle task vectors.
Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.
