Search papers, labs, and topics across Lattice.
This paper investigates Riemannian optimization techniques tailored for low-rank architectures in deep learning, specifically focusing on rank-factored matrix parameters. By examining various geometries and block-matrix configurations, the authors apply their methods to optimize multihead attention parameters in small language models. Although the proposed techniques did not consistently outperform the AdamW baseline after learning rate tuning, the exploration of this design space contributes valuable insights into Riemannian optimization in neural networks.
Riemannian optimization for low-rank architectures reveals intriguing design choices, yet struggles to surpass traditional AdamW performance.
We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications. We examine ten points in the algorithm design space: two geometries for rank-$r$ matrices, three geometries for rank-$r$ partial isometries, and block-matrix variants of these five, where factors are shared across block-rows and block-columns. We apply our methods to the multihead attention parameters in small language models. After tuning learning rates, our methods do not conclusively outperform an AdamW baseline. Our implementations are available online.
