Search papers, labs, and topics across Lattice.
This paper compares amortized point estimators (trained via maximum likelihood or MAP) with amortized posterior inference methods (using normalizing flows, score-based diffusion, or diagonal Gaussian approximations) in the context of in-context learning. The study rigorously evaluates both in-distribution and out-of-distribution generalization across various problem settings, including linear models and shallow neural networks. The key finding is that amortized point estimators generally outperform posterior inference methods, although posterior inference remains competitive in low-dimensional problems.
Forget Bayesian bells and whistles: in-context learning shines brightest with simple point estimators, outperforming complex posterior approximations in most scenarios.
Bayesian and frequentist inference are two fundamental paradigms in statistical estimation. Bayesian methods treat hypotheses as random variables, incorporating priors and updating beliefs via Bayes' theorem, whereas frequentist methods assume fixed but unknown hypotheses, relying on estimators like maximum likelihood. While extensive research has compared these approaches, the frequentist paradigm of obtaining point estimates has become predominant in deep learning, as Bayesian inference is challenging due to the computational complexity and the approximation gap of posterior estimation methods. However, a good understanding of trade-offs between the two approaches is lacking in the regime of amortized estimators, where in-context learners are trained to estimate either point values via maximum likelihood or maximum a posteriori estimation, or full posteriors using normalizing flows, score-based diffusion samplers, or diagonal Gaussian approximations, conditioned on observations. To help resolve this, we conduct a rigorous comparative analysis spanning diverse problem settings, from linear models to shallow neural networks, with a robust evaluation framework assessing both in-distribution and out-of-distribution generalization on tractable tasks. Our experiments indicate that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems, and we further discuss why this might be the case.
Mila