Search papers, labs, and topics across Lattice.
This paper analyzes the scaling exponents of current Large Language Models (LLMs) and argues that their smallness signals an unsustainable trajectory concerning energy consumption. The authors challenge the notion that this smallness can be solely attributed to a numerical bias known as the "pedestal effect," asserting that the unsustainability issue persists even when accounting for this bias. Additionally, they draw parallels between the scaling behavior of LLMs and fluid turbulence, suggesting that data characteristics significantly influence these exponents.
The small scaling exponents of LLMs reveal a troubling energy unsustainability that persists even when accounting for biases in loss function calculations.
We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (``pedestal effect") does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence.
