Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the trade-off between fine-grained valuation and transferability in meta-learning for training-data selection (MTS) by proposing a scalable meta-network approach. Existing MTS objectives struggle with direct meta-network integration due to optimization instability and reliance on superficial features, leading to poor generalization. The authors introduce Transferable Example Scoring and Selection (TESS) with a Pointwise Value Matching (PVM) objective, achieving strong transferability across datasets and model scales for LLM safety and instruction tuning.
A novel meta-network framework for data selection achieves unprecedented transferability across diverse LLM training tasks by overcoming the limitations of existing MTS objectives.
Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valuation and transferability to unseen data. A natural solution is to replace per-sample weights with a selection network. However, we find that directly incorporating such a network into existing MTS objectives leads to unstable optimization and poor generalization, caused by weight suppression and persistent reliance on easy-to-learn features. To address these issues, we propose Transferable Example Scoring and Selection (TESS), a scalable data-selection framework built on a Pointwise Value Matching objective (PVM). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, from subsets to full corpora, and from smaller to larger models.