Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
While frontier AI development heavily depends on scaling laws to optimize compute and architecture, the statistical protocols used to acquire scaling data and extrapolate fits have lacked rigorous evaluation. To resolve this, the authors construct ScAn-Bench, a surrogate benchmarking suite leveraging 4,524 LLM and 8,024 VLM training checkpoints to systematically stress-test scaling methodology across modalities. Their benchmark enables the first standardized assessment of how choices in compute sampling budgets, fitting objectives, and functional forms impact the predictive fidelity of downstream scaling prescriptions.
Frontier pretraining runs routinely stake tens of millions of dollars on scaling laws, yet the empirical pipelines used to fit and extrapolate those curves have never been systematically benchmarked until now.
Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.