Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We identify a novel "accuracy well" phenomenon in deep learning, where classification accuracy exhibits non-monotone dynamics, rising, dropping, and then recovering during training, even with vanishing initialization. This behavior is observed across various network architectures, datasets, and optimizers, suggesting a fundamental aspect of learning dynamics not captured by loss alone. A three-stage explanation involving target-driven organization, a degeneracy bottleneck, and subsequent recovery is proposed and theoretically supported for linear networks.
Deep learning models can achieve near-perfect accuracy, then forget it all, only to relearn it later, defying intuition about monotonic learning curves.
We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.