Search papers, labs, and topics across Lattice.
This paper introduces CausalMix, a novel approach that reformulates data mixture optimization in Large Language Model (LLM) training as a causal inference problem, addressing the limitations of existing methods that rely on static data distributions. By employing a causal model to estimate the Conditional Average Treatment Effect (CATE) across multiple training runs, CausalMix dynamically infers optimal data mixtures that adapt to shifts in the underlying data pool. Extensive experiments demonstrate that models trained with CausalMix significantly outperform traditional methods like RegMix across various downstream tasks, showcasing its effectiveness and interpretability in optimizing LLM performance.
CausalMix reveals that dynamic data mixture optimization can significantly enhance LLM performance, adapting seamlessly to changing data distributions without the need for costly retraining.
In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance. Recent methods optimize mixture weights via proxy models, but they rely on the assumption of static data distributions. As a result, when the underlying data pool shifts, these methods require costly retraining from scratch. This limitation restricts their ability to scale seamlessly from small settings to larger data pools and model sizes. In this paper, we propose CausalMix to address this limitation by casting data mixture optimization as a causal inference problem. We formulate the statistical features of the data pool as covariates and the domain mixture as the treatment. After fitting a causal model on 512 runs of Qwen2.5-0.5B to estimate the Conditional Average Treatment Effect (CATE), we extrapolate the optimal mixture for an 800K data pool and apply it to train a 7B model. Furthermore, we successfully generalize the framework to long chain-of-thought data on Qwen3-4B-Base. By leveraging causal modeling to isolate confounding biases, CausalMix dynamically infers state-dependent optimal data mixtures. Extensive experiments show that the mixture guided by CausalMix consistently improves performance across multiple downstream tasks, outperforming RegMix and other baselines. In addition, we use the CATE Interpreter to provide visual analysis of the learned mixing strategy. Overall, CausalMix offers a causal and interpretable framework for optimizing LLM data mixtures.
Tsinghua AI