Search papers, labs, and topics across Lattice.
This paper provides a theoretical analysis of how Transformers can approximate posterior predictive distributions (PPDs) for Gaussian process regression through in-context learning, a technique known as Prior-data Fitted Networks (PFNs). By constructing a Transformer architecture that implements gradient descent on the PPD's mean and variance, followed by nonlinear mappings, the authors derive error bounds related to attention depth and bin resolution. The analysis highlights the importance of normalization and attention depth for extrapolation beyond the pretraining data range, which is validated through simulations.
Transformers can provably learn Bayesian posterior predictive distributions in-context, revealing the architectural inductive biases that enable this capability.
Prior-data fitted networks (PFNs) have recently emerged as a powerful approach for Bayesian prediction tasks, approximating the posterior predictive distribution (PPD) through in-context learning. Despite their strong empirical performance and ability to go beyond point predictions, theoretical understandings of the algorithmic capability of transformers to learn distributions in context are still lacking. Focusing on Gaussian process regression problems, we show by construction that transformers can implement a gradient descent algorithm targeting the posterior predictive mean and variance, followed by nonlinear mappings that yield binned probabilities of PPD. We study the error bounds of the approximated PPD in terms of attention depth and bin resolution. Based on these results, we further demonstrate the key role of normalization and the choice of attention depth in enabling the extrapolation abilities of transformers beyond the pretraining sample size range. We conduct simulations that corroborate our findings, providing insight into the expressivity of PFNs targeting PPDs and how architectural choices may influence generalization capabilities.
