Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the asymptotic learning dynamics of online softmax classification in a teacher-student setup, revealing a boundary-layer mechanism that leads to a suboptimal α^{-1/3} power-law scaling for generalization error. They show that only examples near the teacher's decision boundaries contribute significantly to learning at late times, while gradient descent noise maintains a non-zero student variance. By optimizing the learning rate schedule, the generalization error can be improved to α^{-1/2}, approaching the Bayes-optimal rate.
Softmax classification's slow α^{-1/3} generalization scaling isn't just about spectral properties, but also stems from a boundary-layer effect where only data near decision boundaries drive learning.
Hard-label classification is usually trained with smooth surrogate losses, most prominently softmax cross-entropy. We isolate an asymptotic mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves in an online teacher-student model. After subtracting the mean logit, the thermodynamic-limit dynamics close in centered variables: a growing centered student-teacher alignment $D$ and the residual student variance $Δ$. At late times, examples away from teacher decision boundaries are already classified confidently and contribute exponentially little. Only boundary layers of width $O(D^{-1})$ remain active, while the noise of fixed-learning-rate online gradient descent maintains a nonzero $Δ$. As a function of the training time $α$ the late-time solution yields a $α^{-1/3}$ power law not only for the test loss but also for the generalization error $ε_g$, i.e., one minus test accuracy. This is much slower than the $α^{-1}$ Bayes-optimal reference for the same model. We further show that learning-rate schedules can improve the generalization error towards a $ε_g \sim α^{-1/2}$ power law. Simulations support the predicted order parameter dynamics and learning curves. Controlled experiments with correlated Gaussian inputs and whitened pretrained features show that data structure can dominate transients. Therefore, our result is an asymptotic, complementary mechanism rather than an alternative to spectral explanations of neural scaling laws.