Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of different likelihood distributions on the performance of Bayesian neural networks (BNNs), specifically contrasting the commonly used Gaussian distribution with alternative distributions like Student's t. The authors conduct experiments across various regression tasks using multilayer perceptrons (MLPs) and find that Student's t consistently outperforms Gaussian likelihoods, enhancing predictive accuracy and sometimes reducing training times. These results highlight the importance of re-evaluating likelihood assumptions in BNNs to avoid model misspecification and improve performance.
Student's t likelihood significantly boosts Bayesian neural network performance, outperforming Gaussian distributions across diverse datasets and architectures.
In Bayesian neural networks (BNNs), variational inference is a widely adopted framework for modeling uncertainty in a distributional way, with the evidence lower bound (ELBO) serving as the standard objective function. Several distributions contribute to the ELBO loss, such as the prior, approximated posterior, and likelihood distribution. Typically, these distributions are all approximated by a Gaussian distribution, since it is easy to compute, allows for reparameterized gradients, and provides a closed-form loss for training. However, several works have highlighted that this assumption may not generally hold, posing the risk of model misspecification. Alternative distributions have been proposed for the prior specifically, while the effect of distribution choice on the likelihood distribution remains unexplored. In this work, our aim is to close this gap by investigating whether alternative assumptions for the likelihood distribution can outperform the commonly used Gaussian. We compare several likelihood distribution assumptions, such as skewed or heavy-tailed, across regression tasks on both artificial and real-world datasets using standard multilayer perceptrons (MLPs). Our findings demonstrate that Student's t yields better predictive performance than a Gaussian likelihood distribution, independent of the data distribution and MLP architecture (depth and width). In some cases, Student's t can also lead to shorter training times, while still being easy to implement.