Search papers, labs, and topics across Lattice.
This paper introduces a novel variational formulation for training shallow neural networks, transforming the discrete optimization problem into a continuum framework that leverages $\lambda$-convex functionals in weighted Sobolev spaces. The authors demonstrate that their approach yields globally well-posed and stable solutions with enhanced regularity, allowing for the optimal parameter density to be derived from a single linear system rather than through iterative optimization. Key findings include explicit generalization error controls and a convergence rate of $O(1/N)$ for finite-width networks, bridging the Neural Tangent Kernel and feature-learning paradigms.
Bypassing iterative optimization, this new variational approach allows for direct computation of optimal parameter densities in shallow neural networks.
Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of $\lambda$-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost $C^3$ regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of $1/\alpha$ relative to the regularization parameter, and prove that finite-width networks of size $N$ achieve the continuum optimum at an $O(1/N)$ rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.
