Search papers, labs, and topics across Lattice.
This paper advocates for a shift towards data-centric LLM training, highlighting the limitations of current ad-hoc data preparation and static dataset utilization. It proposes two complementary research directions: building an agent-based automatic data preparation system for robust workflow construction and scalable data management, and developing a unified data-model interaction training system for dynamic data selection, mixing, and reweighting. The authors argue that these approaches can lead to more efficient, adaptive, and performance-aware data utilization in LLM training.
LLM training is bottlenecked by ad-hoc data preparation and static dataset usage, motivating a shift towards automated, dynamic data management systems.
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks. In current practice, LLM training data is often constructed using ad hoc scripts, and there is still a lack of mature, agent-based data preparation systems that can automatically construct robust and reusable data workflows, thereby freeing data scientists from repetitive and error-prone engineering efforts. Moreover, once collected, datasets are often consumed largely in their entirety during training, without systematic mechanisms for data selection, mixture optimization, or reweighting. To address these limitations, we advocate two complementary research directions. First, we propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management. Second, we argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization. Finally, we discuss the remaining challenges and outline promising directions for future research and system development.
