Search papers, labs, and topics across Lattice.
This paper investigates the impact of data mixing strategies on LLM generalization by establishing a theoretical framework connecting gradient dynamics and domain distributions. They address open questions about domain definition, human-model alignment, and the influence of domain weighting. Based on this analysis, they introduce DoGraph, a graph-constrained optimization framework for data reweighting, and demonstrate its competitive performance on GPT-2 models.
LLMs can achieve competitive performance simply by optimizing data mixing strategies as a graph-constrained optimization problem.
Data mixing strategy is essential for large language model (LLM) training. Empirical evidence shows that inappropriate strategies can significantly reduce generalization. Although recent methods have improved empirical performance, several fundamental questions remain open: what constitutes a domain, whether human and model perceptions of domains are aligned, and how domain weighting influences generalization. We address these questions by establishing formal connections between gradient dynamics and domain distributions, offering a theoretical framework that clarifies the role of domains in training dynamics. Building on this analysis, we introduce DoGraph, a reweighting framework that formulates data scheduling as a graph-constrained optimization problem. Extensive experiments on GPT-2 models of varying scales demonstrate that DoGraph consistently achieves competitive performance.
Tsinghua AI