Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce a novel polynomial activation function, $f(z) = z + \lambda z^3$, projected onto the unit sphere with an unnormalized identity path, enabling training of hundred-layer networks on MNIST without normalization. This construction, when analyzed at a small scale, reveals an exactly solvable polynomial system whose properties dictate optimal design, initialization, and training strategies for deep networks. The key result is that understanding the loss landscape of this simple, small-scale model directly informs the successful training of its deep counterpart.
The loss surface of a simple polynomial activation network exactly solves its own training dynamics, revealing a blueprint for training hundred-layer networks without normalization.
What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + λz 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.