Search papers, labs, and topics across Lattice.
This paper introduces the Instruction Data Selection Agent (DataMaster), which automates the selection of instruction data by interpreting user intent through natural language descriptions, thereby eliminating the need for manual configuration. The approach addresses the limitations of existing metrics that fail to generalize across diverse datasets, streamlining the data curation process. Experimental results demonstrate that DataMaster consistently outperforms static baselines and often exceeds the performance of full-pool training across various domains, including math, medical, and code applications.
Automating data selection with DataMaster not only reduces manual effort but also enhances performance across diverse applications, challenging traditional heuristic methods.
Although existing instruction data selection methods have introduced various metrics, the inherent complexity of real-world datasets makes it impractical for any single metric to generalize across all scenarios. Developers are thus often forced to manually inspect data and craft heuristic rules for each new application---a tedious and error-prone process. In this paper, we propose a paradigm shift from manual configuration to automated orchestration via the Instruction Data Selection Agent (DataMaster), which interprets user intent and autonomously composes optimal selection strategies. By allowing users to specify data needs through natural language descriptions, DataMaster simplifies data curation and removes the burden of manual strategy design. Extensive experiments across the math, medical, and code domains show that DataMaster outperforms static baselines in most settings and surpasses full-pool training in a substantial number of cases. The implementation of DataMaster and the scripts needed to reproduce the reported pipeline are publicly available at https://github.com/nju-websoft/DataMaster.
