Search papers, labs, and topics across Lattice.
This study investigates the potential of generalist coding agents to automate the labor-intensive process of data curation in AI development. By introducing *Curation-Bench*, a benchmark that allows agents to interact with data and implement policies within a fixed training and evaluation framework, the authors demonstrate that agents can achieve competitive data-selection performance. However, the findings reveal an execution-research gap, where agents primarily refine existing policies rather than innovate new ones, highlighting the necessity of scaffolded guidance for effective exploration and adaptation in data research.
Agents can automate the data curation loop but struggle to innovate without structured guidance, revealing a critical gap in their capabilities.
Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.
