Search papers, labs, and topics across Lattice.
This paper introduces FASTMIX, a framework that automates the discovery of optimal data mixtures for model training by reformulating the mixture selection as a bilevel optimization problem. By jointly optimizing mixture coefficients and model parameters through a gradient-based approach, FASTMIX significantly enhances efficiency and scalability compared to traditional methods. Experimental results demonstrate that FASTMIX outperforms existing baselines in both pre-training and post-training scenarios while substantially lowering search costs.
Optimizing data mixtures through gradient descent can drastically reduce training costs and improve model performance without the need for extensive simulations.
While large and diverse datasets have driven recent advances in large models, identifying the optimal data mixture for pre-training and post-training remains a significant open problem. We address this challenge with FASTMIX, a novel framework that automates data mixture discovery while training only a single proxy model. Instead of relying on predefined heuristics or resource-intensive simulations, FASTMIX jointly optimizes mixture coefficients and model parameters, substantially improving efficiency and scalability over prior approaches. At the core of FASTMIX is a reformulation of mixture selection as a bilevel optimization problem. Under this reformulation, we show that optimizing mixture ratios is mathematically equivalent to assigning per-source loss weights under uniform source sampling. This embeds the mixture coefficients directly into the differentiable iterative optimization objective, enabling efficient, gradient-based optimization of both mixture and model. To solve the optimization problem, FASTMIX implements an approximate iterative optimization procedure, alternating between (i) updating model parameters on data sampled according to current mixture ratios (inner loop) and (ii) updating mixture ratios based on validation feedback (outer loop). Across pre- and post-training, FASTMIX outperforms baselines while drastically reducing search cost. Code (https://github.com/hrtan/fastmix)
