Search papers, labs, and topics across Lattice.
This paper advances the understanding of the computational complexity involved in optimally training neural networks, specifically focusing on linear and ReLU activation functions. The authors achieve novel algorithmic upper bounds that establish polynomial-time tractability for ReLU networks with hidden neurons having an out-degree of 1, and they introduce the first non-trivial polynomial-time solvable class for linear networks based on a new data throughput condition. These findings significantly enhance the tractability landscape for neural network training, moving beyond previous limitations in the field.
Polynomial-time training is now achievable for a broader class of neural networks, including those with ReLU activation and specific structural constraints.
In spite of the fundamental role of neural networks in contemporary machine learning research, our understanding of the computational complexity of optimally training neural networks remains incomplete even when dealing with the simplest kinds of activation functions. Indeed, while there has been a number of very recent results that establish ever-tighter lower bounds for the problem under linear and ReLU activation functions, less progress has been made towards the identification of novel polynomial-time tractable network architectures. In this article we obtain novel algorithmic upper bounds for training linear- and ReLU-activated neural networks to optimality which push the boundaries of tractability for these problems beyond the previous state of the art. In particular, for ReLU networks we establish the polynomial-time tractability of all architectures where hidden neurons have an out-degree of $1$, improving upon the previous algorithm of Arora, Basu, Mianjy and Mukherjee. On the other hand, for networks with linear activation functions we identify the first non-trivial polynomial-time solvable class of networks by obtaining an algorithm that can optimally train network architectures satisfying a novel data throughput condition.
