Search papers, labs, and topics across Lattice.
This paper empirically investigates the scaling behavior of in-context learning (ICL) in transformers for Gaussian-mixture binary classification tasks. By systematically varying input dimension, number of in-context examples, and number of pre-training tasks, the authors map the conditions under which ICL succeeds or fails. They also characterize the emergence of benign overfitting in noisy ICL scenarios and its dependence on data geometry and training.
Forget scaling laws: this study reveals a detailed empirical map of *when* and *why* transformers succeed or fail at in-context learning, highlighting the crucial interplay of dimensionality, signal strength, and contextual information.
Transformers have demonstrated a strong ability for in-context learning (ICL), enabling models to solve previously unseen tasks using only example input output pairs provided at inference time. While prior theoretical work has established conditions under which transformers can perform linear classification in-context, the empirical scaling behavior governing when this mechanism succeeds remains insufficiently characterized. In this paper, we conduct a systematic empirical study of in-context learning for Gaussian-mixture binary classification tasks. Building on the theoretical framework of Frei and Vardi (2024), we analyze how in-context test accuracy depends on three fundamental factors: the input dimension, the number of in-context examples, and the number of pre-training tasks. Using a controlled synthetic setup and a linear in-context classifier formulation, we isolate the geometric conditions under which models successfully infer task structure from context alone. We additionally investigate the emergence of benign overfitting, where models memorize noisy in-context labels while still achieving strong generalization performance on clean test data. Through extensive sweeps across dimensionality, sequence length, task diversity, and signal-to-noise regimes, we identify the parameter regions in which this phenomenon arises and characterize how it depends on data geometry and training exposure. Our results provide a comprehensive empirical map of scaling behavior in in-context classification, highlighting the critical role of dimensionality, signal strength, and contextual information in determining when in-context learning succeeds and when it fails.
