Search papers, labs, and topics across Lattice.
This study investigates how incremental grow-and-optimize strategies in deep neural networks can bias training towards flatter regions of the loss landscape by treating growth as progressive constraint relaxation. By starting with a low-dimensional submodel and iteratively expanding the trainable parameters while freezing orthogonal complements, the authors demonstrate that this approach enhances the accessibility of wide basins and suppresses sharp minima. Empirical validation shows that while progressive growth leads to flatter solutions, it does not consistently improve test performance, revealing complexities in the relationship between flatness and generalization.
Progressive growth strategies can significantly bias neural network training towards flatter loss landscapes, but flatter does not always mean better performance.
Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.
