Search papers, labs, and topics across Lattice.
This paper investigates the ability of Transformers to perform in-context learning for linear regression when key assumptions like i.i.d. data, Gaussian noise, and Gaussian regression coefficients are relaxed. They compare Transformer performance against maximum-likelihood estimators under various distributional shifts, including non-Gaussian coefficients, heavy-tailed noise, and non-i.i.d. prompts. The key finding is that Transformers consistently match or outperform these classical baselines, demonstrating robustness to distributional uncertainty in in-context learning.
Transformers can nail in-context learning for regression even when the data is a mess of non-Gaussian noise, heavy tails, and non-i.i.d. distributions, outperforming classical estimators.
Recent work has shown that Transformers can perform in-context learning for linear regression under restrictive assumptions, including i.i.d. data, Gaussian noise, and Gaussian regression coefficients. However, real-world data often violate these assumptions: the distributions of inputs, noise, and coefficients are typically unknown, non-Gaussian, and may exhibit dependency across the prompt. This raises a fundamental question: can Transformers learn effectively in-context under realistic distributional uncertainty? We study in-context learning for noisy linear regression under a broad range of distributional shifts, including non-Gaussian coefficients, heavy-tailed noise, and non-i.i.d. prompts. We compare Transformers against classical baselines that are optimal or suboptimal under the corresponding maximum-likelihood criteria. Across all settings, Transformers consistently match or outperform these baselines, demonstrating robust in-context adaptation beyond classical estimators.
