Search papers, labs, and topics across Lattice.
CORAL, a novel online optimization method, is introduced to efficiently discover near-optimal hardware configurations for deep learning inference on edge devices without offline profiling. It uses distance covariance to capture non-linear dependencies between hardware settings (DVFS, concurrency) and performance metrics, explicitly formulating a throughput-power co-optimization problem. Experiments on NVIDIA Jetson devices with object detection models show CORAL achieves 96-100% of optimal performance in single-target scenarios and consistently finds proper configurations under strict dual-constraint scenarios where baselines fail.
Achieve near-optimal power-efficient deep learning inference on edge devices without the need for expensive and repeated offline profiling, thanks to a novel online optimization method.
For deep learning inference on edge devices, hardware configurations achieving the same throughput can differ by 2$\times$ in power consumption, yet operators often struggle to find the efficient ones without exhaustive profiling. Existing approaches often rely on inefficient static presets or require expensive offline profiling that must be repeated for each new model or device. To address this problem, we present CORAL, an online optimization method that discovers near-optimal configurations without offline profiling. CORAL leverages distance covariance to statistically capture the non-linear dependencies between hardware settings, e.g., DVFS and concurrency levels, and performance metrics. Unlike prior work, we explicitly formulate the challenge as a throughput-power co-optimization problem to satisfy power budgets and throughput targets simultaneously. We evaluate CORAL on two NVIDIA Jetson devices across three object detection models ranging from lightweight to heavyweight. In single-target scenarios, CORAL achieves 96% $\unicode{x2013}$ 100% of the optimal performance found by exhaustive search. In strict dual-constraint scenarios where baselines fail or exceed power budgets, CORAL consistently finds proper configurations online with minimal exploration.
