Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the decision-theoretic optimality of Bayesian Neural Networks (BNNs) in the normal location model under quadratic loss. It demonstrates that BNNs with fixed prior scales produce Bayes decision rules that are not minimax. The authors then introduce a hyperprior on the output variance, resulting in a superharmonic square-root marginal density, which ensures both admissibility and minimaxity of the BNN's decision rule, extending the analysis to predictive density estimation with KL loss.
BNNs, despite their probabilistic appeal, can be statistically sub-optimal unless you carefully tune their prior variance with a superharmonic hyperprior.
Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.