Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper develops a theoretical framework for understanding the competition between in-context learning and specialized prediction in sequence models, using a solvable topic model that mirrors pretraining data heterogeneity. The theory predicts that in-context learning emerges at lower sample complexities, while specialized prediction requires significantly more data, specifically when training tokens exceed the data-generating process's parameters. Empirical evaluation with trained transformers validates these theoretical predictions, demonstrating remarkable agreement in the emergence scales and crossover point of both learning strategies.
In-context learning emerges significantly earlier than specialized prediction, revealing a fundamental trade-off in how LLMs adapt to novel data based on pretraining scale.
Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of \textit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.