Search papers, labs, and topics across Lattice.
This paper introduces a pseudo-Langevin (pL) dynamics method for efficient Boltzmann sampling of feed-forward neural networks trained with large datasets using minibatches. The method leverages the statistical properties of minibatch gradient noise and adjusts fictitious masses and friction coefficients to ensure efficient sampling of the desired equilibrium distribution. Numerical validation demonstrates that pL dynamics maintains high computational diffusion and scales favorably to networks with over one million parameters, outperforming hybrid Monte Carlo (hMC).
Forget full-batch gradients: this new pseudo-Langevin dynamics method lets you efficiently sample the parameter space of large neural networks using minibatches, scaling to millions of parameters.
Sampling the parameter space of artificial neural networks according to a Boltzmann distribution provides insight into the geometry of low-loss solutions and offers an alternative to conventional loss minimization for training. However, exact sampling methods such as hybrid Monte Carlo (hMC), while formally correct, become computationally prohibitive for realistic datasets because they require repeated evaluation of full-batch gradients. We introduce a pseudo-Langevin (pL) dynamics that enables efficient Boltzmann sampling of feed-forward neural networks trained with large datasets by using minibatches in a controlled manner. The method exploits the statistical properties of minibatch gradient noise and adjusts fictitious masses and friction coefficients to ensure that the induced stochastic process samples efficiently the desired equilibrium distribution. We validate numerically the approach by comparing its equilibrium statistics with those obtained from exact hMC sampling. Performance benchmarks demonstrate that, while hMC rapidly becomes inefficient as network size increases, the pL scheme maintains high computational diffusion and scales favorably to networks with over one million parameters. Finally, we show that sampling at intermediate temperatures yields optimal generalization performance, comparable to SGD, without requiring a validation set or early stopping procedure. These results establish controlled minibatch Langevin dynamics as a practical and scalable tool for exploring and exploiting the solution space of large neural networks.
