Search papers, labs, and topics across Lattice.
This study investigates the potential of early training telemetry鈥攕uch as per-epoch loss, training accuracy, and gradient signal-to-noise ratio鈥攖o predict the outcomes of deep neural network training runs. By analyzing 23,788 training runs across various architectures and datasets, the authors demonstrate that gradient-boosted trees can achieve high predictive accuracy for final test accuracy and relative performance using only the first five epochs of data. The findings indicate that useful predictions can be made as early as the first epoch, highlighting the value of early telemetry in optimizing compute resource allocation and guiding training decisions.
Early training telemetry can predict deep learning outcomes with over 99% accuracy after just one epoch, potentially saving significant compute resources.
Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.
