Search papers, labs, and topics across Lattice.
This paper introduces DataOrchestra, a novel framework for orchestrating per-example curation of pretraining data for Large Language Models (LLMs). By adapting the data processing strategy to the specific needs of each example, DataOrchestra integrates various operations such as dropping, cleaning, and rewriting, resulting in improved model performance. The experimental results show that models pretrained with DataOrchestra achieve consistent gains across 11 benchmarks while also reducing computational costs by avoiding unnecessary processing steps.
Tailoring pretraining data processing to individual examples can lead to significant performance improvements in LLMs while cutting down on compute costs.
Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.
