Search papers, labs, and topics across Lattice.
This paper introduces DataPrep-Bench, a comprehensive benchmark designed to evaluate the dual capabilities of large language models (LLMs) in training data preparation: data construction and data quality evaluation. By employing a downstream-grounded protocol across six domains, the benchmark assesses how well LLMs transform raw sources into supervised training data and predict the training utility of candidate datasets. Key findings reveal that the newly proposed Distributional Alignment Score (DAS) significantly outperforms existing evaluators, achieving strong correlations with downstream performance, particularly in Math, Science, and Medical domains.
LLMs can now be benchmarked for their ability to prepare training data, revealing that a new evaluation metric outperforms traditional methods in predicting downstream utility.
The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout,"quality"refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r>0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.
