Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper reinterprets gradient descent in feed-forward ReLU networks as a collective dynamics in training-set space rather than in weight space, focusing on the behavior of residuals. By eliminating weight variables for single hidden layers, the authors derive a closed equation governed by a collective kernel, which is further extended to deeper networks through a hierarchy of weight-induced Gram operators. The findings reveal a structured layer-wise kernel that facilitates information transport, offering insights into the dynamics of deeper networks beyond just weight adjustments.
A novel perspective on gradient descent reveals that deeper networks require a hierarchy of Gram operators for effective information transport, challenging traditional views on weight dynamics.
We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers.