Search papers, labs, and topics across Lattice.
This paper reveals that Mean Field Variational Inference (MFVI), traditionally viewed as underestimating posterior variance, can paradoxically overestimate predictive variance in certain directions, particularly where training data is concentrated. Through the analysis of conjugate Bayesian Linear Regression, the authors demonstrate that while MFVI may underestimate predictive variances in some areas, it compensates by overestimating them in others, leading to an unexpected increase in expected predictive variance for test points drawn from the training distribution. The findings highlight a critical relationship between MFVI's behavior and the Cold Posterior Effect, suggesting that adjusting the temperature can mitigate this overestimation and improve predictive accuracy.
MFVI can paradoxically overestimate predictive variance, leading to predictions that exceed those of the exact posterior for in-distribution test points.
Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks.
