Search papers, labs, and topics across Lattice.
This work formulates a statistical-mechanical theory of neural network learning directly in function space by mapping microscopic weight trajectories to macroscopic dynamics governed by the learning operator \(M = JJ^\ast\). Integrating local stochastic fluctuations against the parameter-space density of states curvature \(B\) yields an effective fluctuation potential \(\Phi_{\mathrm{fluc}} \propto \log\det(M^{-1} + B)\) whose free energy is minimized when \(M\) and \(B\) commute and anti-align in spectrum. For ReLU networks, this establishes that dynamical learning forces drive the model to relax preferentially along smooth, low-curvature functional directions over high-frequency noise.
Overparameterized networks are thermodynamically driven toward smooth, data-adaptive solutions by a function-space fluctuation potential that penalizes misalignment between dynamic mobility and parameter-space curvature.
Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator \(M=JJ^\ast\). Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator \(B\), and integrating over local fluctuations yields $$ \Phi_{\mathrm{fluc}}(M;B)=\frac{\sigma_\xi^2}{2}\log\det(M^{-1}+B)+\mathrm{const}. $$ At fixed spectrum, this term is rotationally stationary when \([M,B]=0\), is minimized by pairing large eigenvalues of \(M\) with small eigenvalues of \(B\), and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions, \(B=\sigma_\xi^2L^\ast\mathcal K L\), where \(L\) measures coarse-grained second-order structure. Thus the low-\(B\) sector corresponds, up to bounded anisotropy of \(\mathcal K\), to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.
