Search papers, labs, and topics across Lattice.
Tsinghua AIAffiliation:Beijing National Research Center For InformationJiangsu Normal UniversityAug 24, 2026arXiv:2608.23475This paper introduces StrategyBench, a benchmark designed to evaluate the explicit strategy induction capabilities of large language models (LLMs) in few-shot in-context learning scenarios. By contrasting direct ICL with human-like strategy abstraction and application, the authors demonstrate that the utility of explicit strategies varies significantly across different task categories and is influenced by both the generation and execution conditions of these strategies. The findings reveal critical insights into how LLMs can better adapt to evolving tasks, highlighting the importance of strategy quality for downstream performance.
Explicit strategy induction can dramatically enhance LLM performance, but its effectiveness varies widely across task types and configurations.
As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.
