Search papers, labs, and topics across Lattice.
This paper introduces a framework for modeling the energy consumption of Transformer training on multiple GPUs, addressing the urgent need for sustainable and cost-effective system design in the face of increasing computational demands. By conducting controlled architectural sweeps of BERT models, the authors establish a relationship between measured energy and lightweight proxies for compute, memory traffic, and hardware efficiency. The resulting scaling law model effectively predicts training energy across diverse configurations, incorporating a novel speedup-based hardware-efficiency factor that accounts for tensor and data parallelism.
Energy consumption during Transformer fine-tuning can be accurately predicted across various configurations, revealing critical insights for sustainable AI development.
Transformer-based models underpin modern natural language processing but incur rapidly growing computational and energy costs. As training scales in both model size and parallelism, accurately predicting energy consumption has become critical for sustainable and cost-aware system design. We present a framework for modeling the energy consumption of Transformer training on multiple GPUs. Using controlled architectural sweeps of BERT models, we relate measured energy to lightweight proxies for compute, memory traffic, and hardware efficiency. Inspired by roofline models, our approach incorporates a speedup-based hardware-efficiency factor that captures the effects of tensor parallelism and fully sharded data parallelism. We derive a scaling law model that accurately predicts training energy across heterogeneous configurations.
