Search papers, labs, and topics across Lattice.
This paper investigates targeted instruction selection for fine-tuning LLMs by disentangling the effects of data representation and selection algorithms. Through controlled experiments across models, tasks, and budgets, the authors find that gradient-based data representations are most effective at identifying subsets whose similarity to the query set predicts performance. They also demonstrate that gradient-based representations with greedy round-robin selection perform well at low budgets, and unify existing selection algorithms as approximate distance minimization techniques, providing generalization bounds.
Forget complex selection algorithms: gradient-based data representations are the key ingredient for effective targeted instruction selection in LLMs.
Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets and models. While no single method dominates, gradient-based representations paired with a greedy round-robin selection algorithm tend to perform best on average at low budgets, but these benefits diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.
