Search papers, labs, and topics across Lattice.
This paper introduces CARB, a framework that accurately predicts the inference cost of CNNs鈥攊ncluding energy, latency, and peak memory鈥攂y analyzing a comprehensive workload characterization study of over 13,000 CNN configurations on two GPU platforms. The study reveals significant differences in how energy, latency, and memory scale under varying computational demands, with energy and latency diverging by up to 3x under high loads. CARB achieves an impressive R虏 of ~0.99 in its predictions and employs a two-stage deployment screening that efficiently narrows down design options, validating against actual hardware performance.
Energy and latency can diverge by 3x under high computational demand, necessitating platform-specific models for accurate CNN inference cost predictions.
Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resource-constrained GPU platforms. Existing approaches rely on FLOPs, latency measurements, or single-device profiling as energy proxies, overlooking the non-linear interactions between architectural design and hardware load. We present a workload characterization study of 13 419 CNN configurations on two GPU platforms (RTX 5090 and RTX 3080) under GPU telemetry, revealing that energy, latency, and memory exhibit fundamentally distinct scaling behaviors: energy and latency diverge by 3x under high computational demand, and cross-GPU transferability differs by target--energy and latency require platform-specific models while memory transfers well across the two tested platforms. Building on these characterization findings, we develop CARB, a cascade-blended ensemble that jointly predicts all three targets with R2 ~0.99, and a two-stage deployment screening workflow that eliminates over 90% of candidates in seconds, reducing large design spaces to a Pareto-prioritized shortlist validated against real hardware.
