Search papers, labs, and topics across Lattice.
This paper introduces a novel measure of training data influence that operates independently of downstream tasks or validation sets, addressing the challenges of evaluating language model pretraining. By analyzing the gradient updates from intermediate checkpoints, the authors quantify how much each training example contributes to the convergence of model parameters over time. The findings reveal a systematic shift in data influence, with literature-related data being more impactful early in training and STEM data gaining prominence in later stages, providing insights into the evolving role of different data types throughout the pretraining process.
Training data influence shifts dramatically over the course of language model pretraining, with literature data dominating early and STEM data taking over later.
Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.
