Search papers, labs, and topics across Lattice.
This paper evaluates subspace Levenberg-Marquardt algorithms, specifically Krylov subspace LM (KSLM) and hybrid subspace LM (HSLM), to enhance the efficiency of second-order methods in training neural networks. By comparing these subspace variants against the classical Levenberg-Marquardt method and popular first-order algorithms like stochastic gradient descent and Adam, the authors demonstrate that subspace methods can significantly reduce computational and memory costs while maintaining performance. The key finding reveals that subspace LM approaches achieve competitive results in both regression and classification tasks, making them viable alternatives for larger neural networks.
Subspace Levenberg-Marquardt algorithms can outperform traditional methods in training larger neural networks without sacrificing performance.
The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computational and memory costs increase significantly as the number of parameters in an NN grows. To address this limitation, subspace methods have been proposed, such as the Krylov subspace LM (KSLM) and the hybrid subspace LM (HSLM), making second-order algorithms more efficient. In this work, we evaluate the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks. We compare the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent (SGD) and Adam.
