Search papers, labs, and topics across Lattice.
This study investigates the frequency scaling behavior of NVIDIA GPUs under machine learning workloads, revealing that lower-performance GPUs exhibit significant frequency dependency influenced by recent workload history within an 80ms window. This finding challenges the prevalent assumption in ML latency-prediction models that kernel latencies are independent, highlighting the inter-kernel dependencies introduced by dynamic frequency scaling. The research suggests potential improvements in latency-prediction models and GPU kernel strategies that could enhance performance and efficiency in ML applications.
GPU frequency behavior reveals inter-kernel dependencies that could revolutionize latency-prediction models in machine learning.
This work presents ongoing research on the frequency scaling behavior of NVIDIA GPUs when executing ML/AI workloads. Our preliminary findings show that, on lower-performance GPUs, the operating frequency is strongly affected by the recent workload history, typically within an 80ms window. This behavior challenges a common assumption underlying several state-of-the-art ML latency-prediction techniques, which treat individual GPU kernel latencies as independent and therefore estimate total execution time by summing isolated per-kernel measurements. Our results indicate that such an assumption does not always hold, as the GPU's dynamic frequency scaling introduces inter-kernel dependencies. We also outline several promising directions for leveraging this observation in future work, including improved latency-prediction models, GPU kernel-reordering strategies, and NAS-driven guidelines for frequency/latency/energy-aware model design.
