Search papers, labs, and topics across Lattice.
This paper introduces MOON (Multi-Objective OrthoNormalized Updates), a novel approach for multi-task learning that addresses the limitations of existing gradient manipulation methods by leveraging the matrix structure of model parameters. By applying gradient updates under spectral-nuclear norm geometry, MOON achieves improved optimization efficiency and multi-task performance compared to traditional Euclidean methods. Theoretical convergence rates of $\mathcal{O}(T^{-1/2})$ and $\mathcal{O}(T^{-1/4})$ are established, and empirical evaluations across benchmarks validate its effectiveness.
Gradient manipulation in multi-task learning can be significantly enhanced by recognizing the matrix structure of model parameters, leading to superior optimization outcomes.
Multi-objective optimization (MOO) has demonstrated significant success in multi-task learning by mitigating task conflicts through gradient manipulation. However, most existing methods flatten model parameters into vectors and perform gradient manipulation under Euclidean geometry, thereby overlooking the matrix structure prevalent in modern architectures such as Transformers. In this paper, we show that gradient manipulation in Euclidean space does not generally yield the steepest descent direction under matrix geometry, potentially limiting optimization efficiency. Drawing from the theory of steepest descent for matrix-valued parameters, we propose MOON (Multi-Objective OrthoNormalized Updates), which performs gradient manipulation under spectral--nuclear norm geometry and uses the orthonormalized manipulated gradient for parameter updates. Theoretically, for smooth non-convex objectives, we establish convergence of the averaged Pareto-stationarity measure at rates of $\mathcal{O}(T^{-1/2})$ in the deterministic setting and $\mathcal{O}(T^{-1/4})$ under stochastic gradients. Empirical results across various benchmarks show that MOON consistently improves both optimization efficiency and final multi-task performance. Our code is available at https://github.com/KunlinLyu/MOON.
