Search papers, labs, and topics across Lattice.
This paper introduces a novel analytical framework for estimating the energy consumption of large language model (LLM) inference on NVIDIA H100 GPUs, addressing the limitations of direct measurement methods. By integrating parameter-scaled FLOP accounting with calibrated memory-traffic factors and energy coefficients, the methodology allows for detailed energy estimates across various components of LLM inference. The key finding reveals that this approach enables transparent and reproducible energy assessments, facilitating comparative studies and sustainability evaluations without the need for specialized hardware instrumentation.
Energy estimates for LLM inference can be achieved without direct measurements, revealing insights into the environmental impact of AI systems.
The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.
MIT CSAIL