Search papers, labs, and topics across Lattice.
This paper explores the implications of different performance metrics on the capabilities of AI models, particularly in the context of exponential compute scaling. By classifying metrics based on their relationship to training and inference compute, the authors derive mathematical conditions that reveal how certain bounded metrics favor "meek models," while unbounded metrics can lead to a concentration of capabilities among wealthy developers. The findings underscore the necessity of careful metric selection, as the choice between bounded and unbounded metrics can significantly influence policy decisions and the equitable distribution of AI capabilities.
The choice of performance metrics could determine whether AI capabilities remain concentrated among the wealthy or proliferate across a broader developer base.
As exponential compute scaling continues, will the capabilities of frontier AI models outstrip what is accessible to developers on a small fixed budget? Or will capabilities converge, with "meek models inheriting the earth"? Building on Gundlach et al. (2025b), we show that the answer depends on how we value and measure AI capabilities. We discuss conventional performance measures and show that, while validation loss shows a shrinking gap, on other metrics frontier models grow their lead forever. Classifying performance metrics by their functional forms in relation to training (and inference) compute, we provide tight mathematical conditions for determining which metrics favor meek models, and show that bounded performance metrics always do. But careful interpretation of performance metrics is essential: we show that many common bounded metrics have closely-related counterpart metrics that are unbounded (and vice versa). Determining the apt metric in a domain is a prerequisite for policy, since bounded and unbounded metrics may suggest opposing policy responses. If a particular capability -- like software engineering, synthetic biology, or rhetorical persuasiveness -- is unbounded when measured in the terms we care about, frontier-level capability will likely be concentrated in the hands of a few wealthy actors. Conversely, if that capability is instead bounded, frontier-level capabilities proliferate through meek models into the hands of the many.
MIT CSAIL