Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper empirically analyzes the amortization error of prior-data fitted networks (PFNs) by decomposing deployment error into prior, context, and amortization components, measuring the latter against a reference posterior predictive distribution. While PFNs generally approximate the target PPD well, residual amortization error persists in rare prior settings, even with increased model scale, leading to disagreement between independently pretrained models. Bayesian model averaging effectively mitigates this uncertainty and improves predictive performance.
Rare events under the prior pretraining distribution expose persistent amortization error in prior-data fitted networks, even at scale.
Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.