Search papers, labs, and topics across Lattice.
This paper systematically analyzes the mechanisms of deep feedforward ReLU networks, focusing on the relationships between paths in the network to elucidate the training solutions derived from back-propagation. It reveals that units in deep ReLU networks create piecewise linear manifolds that partition the input space, contrasting with the hyperplane divisions seen in two-layer networks. The findings generalize principles from simpler architectures to deeper networks, providing insights into how hidden-layer units can be effectively utilized for complex function approximations and training solutions.
Deep ReLU networks can create intricate piecewise linear partitions of input space, fundamentally altering our understanding of their training dynamics.
The architecture of deep feedforward neural networks is ubiquitous in deep learning, either as a whole system or as a subnetwork of other architectures, and thus its mechanism is a key ingredient of the black box of neural networks. On the basis of the simplest two-layer ReLU network, this paper systematically studies the mechanism of deep feedforward ReLU networks with multiple hidden layers and successfully explains the training solution obtained by the back-propagation algorithm. The concept of a path, especially in terms of the relationships between paths, plays a central role in uncovering the mystery of the black box. It is shown that a unit of a deep ReLU network can form a piecewise linear manifold to divide the input space, instead of a hyperplane of the two-layer case. How to efficiently use the hidden-layer units to produce both linear functions and partitions of the input space is also a central problem. The principles of a two-layer ReLU network can be generalized to the deeper case to a large extent, such as multiple strict partial orders and continuity restriction. The combination of the basic and simple principles proposed can yield complicated instantiations including the training solutions, and in this sense the black box of deep feedforward ReLU networks is revealed.
