Search papers, labs, and topics across Lattice.
This paper introduces Dirichlet scale mixture (DSM) priors for Bayesian Neural Networks (BNNs) to address limitations such as overconfidence and vulnerability to adversarial attacks. The authors theoretically derive dependence structures and shrinkage results for DSM priors, demonstrating how they induce sparsity in neural networks. Empirical results on simulated and real-world data show that DSM priors encourage sparse networks, improve robustness against adversarial attacks, and achieve competitive predictive performance with fewer parameters, particularly in correlated, moderately small data regimes.
Ditch your Gaussian priors: Dirichlet scale mixture priors offer a principled way to shrink BNNs, improve robustness, and mitigate the cold posterior effect, especially when data is scarce and correlated.
Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.
