Search papers, labs, and topics across Lattice.
This paper introduces a weighted in-context influence (wICI) framework for instruction data selection, aiming to identify high-quality examples that improve instruction-following for related samples. wICI quantifies the impact of each candidate example on reducing the difficulty of semantically similar peers when used in-context. Experiments across models and benchmarks show that wICI-selected data outperforms existing baselines under data constraints, and that sample difficulty is negatively correlated with in-context influence.
Forget data scale, focus on influence: a new metric reveals that the best instruction tuning data isn't necessarily the most obvious or easiest.
Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.
