Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
We introduce a theoretical framework that disentangles the effects of unsupported instructions and target tasks on In-Context Learning (ICL). The framework demonstrates that increasing demonstration count can overcome prompt template sensitivity for supported tasks, and that demonstration evidence can dominate instruction shifts for unsupported instructions under a fixed reference extension. Furthermore, it explains how ICL can solve long-horizon, unsupported tasks by decomposing them into pretraining-supported subtasks, leveraging familiar subtasks to avoid direct selection of the unsupported complete task.
LLMs can learn from instructions and tasks entirely outside their pretraining distribution by decomposing them into known subtasks.
In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates. Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support. We develop a theoretical framework under mild assumptions that separates these two forms of distribution shift. For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases. For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an $N$-independent instruction shift while preserving the exponential identification rate. We then use the same ICL framework as an interpretive lens for long-horizon target tasks that are not represented in pretraining but admit a plan that decomposes them into pretraining-supported subtasks. Applying the supported-task analysis step by step yields a standard accumulated-error bound and clarifies how invoking familiar subtasks along an intermediate-state trajectory can avoid directly selecting the unsupported complete task. Together, our results characterize when demonstrations, instructions, and task decomposition can support ICL beyond the corresponding pretraining supports.
Microsoft Research