Search papers, labs, and topics across Lattice.
This paper introduces the Skaling law, a novel framework that couples model capacity and training data through a single interaction exponent, addressing the limitations of traditional neural scaling laws that misestimate loss in data-scarce and overtraining scenarios. By applying this new law, the authors achieve a 1.5-3x reduction in Mean Absolute Percentage Error (MAPE) across various training regimes. Additionally, when combined with a sparse grid strategy, the Skaling law allows for accurate extrapolation with approximately 10x less compute, enhancing resource efficiency in model training.
The Skaling law slashes loss estimation errors by up to 3x and reduces compute needs by 90% for reliable model performance predictions.
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.
Meta AI