Search papers, labs, and topics across Lattice.
This paper operationalizes the concept of epiplexity as a guiding principle for data selection and synthetic data generation to enhance out-of-distribution generalization in machine learning models. By fitting scaling laws to training loss curves, the authors adaptively adjust sampling weights over domains to maximize expected epiplexity gains during training. The results demonstrate that higher epiplexity correlates with improved performance on zero-shot and fine-tuning tasks, validating the hypothesis that structurally rich data enhances cross-domain representation transfer.
Higher epiplexity in training data can significantly boost model performance in unseen tasks, revealing a new avenue for data-driven generalization strategies.
Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that could be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, provides a mechanism to reason about this relationship. In this paper, we show how to operationalize epiplexity as an online training signal for data selection and synthetic data generation. For selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, we define a generator's reward as the change in learner epiplexity over a buffer of previously generated data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning based tasks, supporting the hypothesis that data rich in structural information yield representations that transfer across domains.
