Search papers, labs, and topics across Lattice.
This paper analyzes wide Bayesian neural networks using large deviation theory to characterize the statistically dominant, non-Gaussian fluctuations that determine posterior concentration. They derive a variational objective (rate function) on predictors that incorporates an emerging notion of complexity and feature learning at the functional level. The analysis reveals that the posterior output rate function requires joint optimization over predictors and internal kernels, moving beyond the fixed-kernel assumption of Neural Network Gaussian Process (NNGP) theory.
Bayesian neural networks actually learn features in the large width limit, defying the conventional wisdom of fixed-kernel NNGP theory.
We study wide Bayesian neural networks focusing on the rare but statistically dominant fluctuations that govern posterior concentration, beyond Gaussian-process limits. Large-deviation theory provides explicit variational objectives-rate functions-on predictors, providing an emerging notion of complexity and feature learning directly at the functional level. We show that the posterior output rate function is obtained by a joint optimization over predictors and internal kernels, in contrast with fixed-kernel (NNGP) theory. Numerical experiments demonstrate that the resulting predictions accurately describe finite-width behavior for moderately sized networks, capturing non-Gaussian tails, posterior deformation, and data-dependent kernel selection effects.
