Search papers, labs, and topics across Lattice.
This paper introduces WattGPU, a novel predictive framework designed to estimate mean GPU power draw and Inter-Token Latency (ITL) for large language model (LLM) inference without the need for exhaustive profiling. By utilizing only publicly available metadata and specifications, WattGPU generalizes effectively to unseen NVIDIA GPUs and LLMs, achieving a median absolute percentage error of ≤3.4% for power draw and ≤8.5% for latency in server scenarios. The results demonstrate a significant improvement over traditional baselines, offering a more efficient approach to optimizing LLM deployments in data centers.
WattGPU slashes prediction errors by up to 4x for unseen LLM-GPU combinations, revolutionizing how we optimize energy consumption in AI inference.
Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce \textit{WattGPU}, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of $\leq3.4\%$ for offline and $\leq13.5\%$ for server scenarios on unseen GPUs, while the latency model achieves $\leq8.5\%$ in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall $\tau\geq0.76$). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4$\times$ on unseen LLM-GPU combinations for server scenarios or approximately 2$\times$ for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.
