Search papers, labs, and topics across Lattice.
This paper analyzes how overparameterization introduces weight-space symmetries in neural networks that improve optimization. It proves that these symmetries diagonally precondition the Hessian, leading to better-conditioned minima within equivalence classes of solutions. Additionally, overparameterization increases the probability mass of global minima near typical initializations.
Overparameterization doesn't just add parameters; it warps the loss landscape geometry itself, making well-conditioned minima easier to find.
Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in two distinct ways. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each equivalence class of functionally identical solutions. Second, we show that overparameterization increases the probability mass of global minima near typical initializations, making these favorable solutions more reachable. Teacher-student network experiments validate our theoretical predictions: as width increases, the Hessian trace decreases, condition numbers improve, and convergence accelerates. Our analysis provides a unified framework for understanding overparameterization and width growth as a geometric transformation of the loss landscape.
