Search papers, labs, and topics across Lattice.
Moving beyond standard Gaussian parameter assumptions, this work formalizes learnability in teacher-student networks by analyzing how the geometric dissimilarity of teacher hidden nodes controls convergence to the global minimum. By analytically reducing the infinite-data loss landscape to two dimensions under a fast-readout limit, the authors prove that high node dissimilarity drives optimization into interior local minima, whereas low dissimilarity traps students in out-of-bounds boundary minima. These geometric insights reveal that the traditional i.i.d. teacher assumption hides severe optimization failure modes, motivating a targeted training intervention鈥攂oosting readout learning rates while damping inner bias rates鈥攖hat consistently recovers global convergence.
Standard Gaussian teacher assumptions hide severe optimization traps: hidden node alignment alone dictates whether gradient descent fails into boundary versus interior local minima.
Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.
