Search papers, labs, and topics across Lattice.
This paper investigates the complexity-theoretic inductive bias of Bayesian neural networks in the infinite-width limit, focusing on the mean-field scaling regime. The authors establish a width-robust learnability theorem, demonstrating that a family of Boolean-cube targets is learnable from polynomially many samples at infinite width if and only if it is also learnable at polynomial width, contingent on the reduced entropy being polynomially bounded. The findings reveal that the mean-field learner can generalize precisely on targets representable by polynomial-size networks, providing a clearer understanding of learning dynamics without the complications of width-dependent generalization power.
Infinite-width Bayesian neural networks can learn complex functions with the same efficiency as their finite-width counterparts, revealing a surprising equivalence in learnability.
Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling. The central quantity is the \emph{reduced entropy} \[ s_\infty(y,\varepsilon)=\limsup_N -\frac{1}{N}\log 蟺_N^0(L\le \varepsilon), \] the intensive prior cost of representing a target function $y$ to population mean-squared error $\varepsilon$. Our main result is a width-robust learnability theorem. At fixed depth, a family of Boolean-cube targets is learnable from polynomially many samples at infinite width if and only if it is learnable at polynomial width, if and only if its reduced entropy is polynomially bounded. Equivalently, up to polynomial slack in accuracy, the Bayesian mean-field learner generalizes exactly on the targets that can be represented by polynomial-size networks. The forward direction is proved by a form of subsampling: from the infinitely many hidden neurons in the mean-field solution, one can select polynomially many representatives and still preserve the learned function on every input simultaneously. At the critical scaling this subsampling has both an ``active'' component, which keeps the data-dependent low-dimensional statistics, and a ``lazy'' component, which resamples the entropy-dominated directions from the prior. Thus the infinite-width mean-field limit gives a clean analytic description of learning without introducing spurious width-dependent generalization power.
