Search papers, labs, and topics across Lattice.
This paper identifies critical power inefficiencies in multimodal model inference serving on GPUs, highlighting issues such as idle inter-stage dependency stalls and the adverse effects of thermal throttling. To address these inefficiencies, the authors introduce Tri-serve, a software-based dynamic voltage and frequency scaling (DVFS) controller that optimally manages power based on inter-stage dependencies, arithmetic intensity, and thermal conditions. Tri-serve achieves a 22% improvement in energy efficiency without compromising latency or throughput, marking a significant advancement in energy management for multimodal inference tasks.
Tri-serve redefines energy efficiency in multimodal inference by addressing hidden power inefficiencies, achieving a 22% boost without latency trade-offs.
Multimodal model inference creates substantial energy demand with growing performance requirements. Within GPUs, power is autonomously managed by an on-board power management unit (PMU), which makes frequency boosting/throttling decisions. However, we find that these hardware-managed frequency decisions can cause significant power inefficiency. This work identifies three classes of power inefficiencies within modern multimodal inference serving: (1) inter-stage dependency stalls run at near maximum frequency despite being idle; (2) anti-correlation between auto-boost frequency and arithmetic intensity (A.I.) results in compute-bound phases (e.g., prefill) running at lower frequency and vice versa; and (3) thermal throttling degrades SM frequency and throughput. We propose Tri-serve, a software-based DVFS controller that jointly accounts for three classes of inefficiency -- inter-stage Dependency stalls, the Arithmetic-intensity effect on frequency and power, and the Thermal-throttling effect of high A.I. phases -- to deliver energy-efficient multimodal serving on commodity GPUs. We show that Tri-serve achieves 22% energy efficiency improvement with no latency or throughput impacts.
NVIDIA