Search papers, labs, and topics across Lattice.
This paper introduces the Unified Neural Scaling Law (UNSL), a novel functional form that models the simultaneous scaling behaviors of deep neural networks across multiple dimensions, including model parameters, dataset size, training steps, and compute. The significance of this work lies in its ability to provide more accurate extrapolations of scaling behavior compared to existing models across a diverse range of tasks, including vision, language, math, and reinforcement learning. Key results indicate that UNSL outperforms traditional scaling laws, offering a more reliable framework for predicting the performance of neural networks as they scale.
UNSL reveals that accurately modeling neural network scaling across multiple dimensions can significantly enhance performance predictions, outperforming traditional scaling laws.
We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. This set includes large-scale vision, language, math, and reinforcement learning. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.
Mila