Search papers, labs, and topics across Lattice.
This study investigates the in-context learning capabilities of large language models (LLMs) by comparing the effectiveness of learning from rules versus examples across five diverse tasks. The findings reveal that LLMs generally learn more reliably from rule-based instructions than from example-based prompts, with the advantage of rule learning being particularly pronounced in algebraic tasks. Additionally, while instruction tuning enhances the benefits of rule-based learning, simply increasing the number of examples does not yield consistent improvements in performance.
LLMs achieve more reliable in-context learning from rules than from examples, challenging the conventional reliance on few-shot prompting.
Large language models (LLMs) exhibit in-context learning capabilities, where they can learn new tasks from prompt contexts without weight updates. We compare the learning efficacies of two prominent modes of in-context learning: (1) learning from descriptions of rules (instruction following); and (2) learning from examples of input-output demonstrations (few-shot prompting). Through five learning tasks that cover diverse domains (games, arithmetic, linguistic inferences), we compare two modes of learning (rules vs. examples) specifying the same underlying task. We furthermore explore model and task properties that modulate the learning efficacies. We find that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains. Instruction tuning amplifies the benefit of rule-based learning while keeping example-based learning capacities intact. Surprisingly, we find no privileged effect of example-based learning in base models, and rules still lead to gains in algebraic task domains. Overall, the comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity and/or recruits parametric knowledge.
