Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper benchmarks the energy consumption of 14 LLMs (7B-72B parameters) on the MMLU benchmark across five subjects, measuring CO2 emissions using the Perun framework on NVIDIA A100 GPUs. The study finds a strong positive correlation between model size, reasoning capabilities, token generation, and CO2 emissions, with larger, reasoning-enabled models achieving higher accuracy (up to 84.9%) at the cost of significantly increased energy usage. Subject-level analysis reveals that symbolic domains like Abstract Algebra are particularly computationally expensive and yield lower accuracy, underscoring the need for more efficient reasoning strategies.
Reasoning with LLMs comes at a steep carbon cost: larger, more accurate models can emit significantly more CO2, especially on abstract tasks.
This study presents a comprehensive evaluation of the environmental cost of large language models (LLMs) by analyzing their performance, token usage, and CO2 equivalent emissions across 14 LLMs ranging from 7 to 72 billion parameters. Each LLM was tasked with answering 500 multiple-choice and 500 free-response questions from the MMLU benchmark, covering five diverse subjects. Emissions were measured using the Perun framework on an NVIDIA A100 GPU and converted through an emission factor of 480 gCO2/kWh. Our results reveal strong correlations between LLM size, reasoning behavior, token generation, and emissions. While larger and reasoning-enabled models achieve higher accuracy, up to 84.9%, they also incur substantially higher emissions, driven largely by increased token output. Subject-level analysis further shows that symbolic and abstract domains such as Abstract Algebra consistently demand more computation and yield lower accuracy. These findings highlight the trade-offs between accuracy and sustainability, emphasizing the need for more efficient reasoning strategies in future LLM developments.