Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a systematic procedure for deriving robust scaling laws and comparing architectural choices in High-Energy Physics (HEP) machine learning models, building on observed scaling behaviors similar to those in language models. The methodology validates scaling trajectories on toy problems before applying them to multi-task transformers on the ATLAS JetSet2 dataset, predicting jointly optimal hyperparameters under compute- and data-constrained regimes. Key findings include a $\sqrt{C}$ dependence for model and dataset size at compute-optimal scaling, the benefit of auxiliary objectives, and the critical role of large, high-quality datasets for establishing the power-law regime.
Large HEP datasets are not just for training; they are foundational for unlocking power-law scaling, with smaller datasets offering little insight into high-compute performance.
Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks. We first validate the full scaling trajectory on toy problems and then apply the procedure to multi-task transformers on the ~11 billion-jet ATLAS JetSet2 dataset, in both the compute- and data-constrained regimes. For the latter, we predict, to the best of our knowledge for the first time, the jointly optimal model size, training horizon, learning rate and batch size under early stopping. At compute-optimal scaling, we recover a near-equal $\sqrt{C}$ dependence of model and dataset size, and find that auxiliary objectives lower the primary jet-classification loss at equal compute budget. Expanding the inputs toward lower-level data systematically lowers the loss while leaving the scaling exponent nearly unchanged. The onset of the power-law regime is itself set by scale: below a threshold in dataset size the loss carries little information about high-compute scaling, underscoring the value of large, high-quality full-simulation datasets as a foundation for scaling studies and the development of foundation models in HEP.
TU Munich