Search papers, labs, and topics across Lattice.
This paper introduces a latency-oriented neural network learning method specifically designed for edge systems, addressing the challenge of optimizing deep learning models under strict latency constraints. By employing a universal hardware-customized latency predictor, the authors achieve high accuracy while adhering to real-time performance requirements, demonstrating significant reductions in inference latency for popular architectures like GoogLeNet and VGG-19. Experimental results show that their approach can maintain competitive accuracy while meeting stringent latency targets, outperforming existing methods in both latency reduction and accuracy improvement.
Achieving real-time performance on edge devices, this method reduces GoogLeNet's latency from 40.32 ms to 34 ms with minimal accuracy loss, while VGG-19 sees a latency drop from 119.98 ms to 34 ms with an accuracy gain.
Deep learning applications have been widely adopted on edge devices, to mitigate the privacy and latency issues of accessing cloud servers. Deciding the number of neurons during the design of a deep neural network to maximize performance is not intuitive. Particularly, many application scenarios are real-time and have a strict latency constraint, while conventional neural network optimization methods do not directly change the temporal cost of model inference for latency-critical edge systems. In this work, we propose a latency-oriented neural network learning method to optimize models for high accuracy while fulfilling the latency constraint. For efficiency, we also introduce a universal hardware-customized latency predictor to optimize this procedure to learn a model that satisfies the latency constraint by only a one-shot training process. The experiment results reveal that, compared to state-of-the-art methods, our approach can well-fit the'hard'latency constraint and achieve high accuracy. Under the same training settings as the original model and satisfying a 34 ms latency constraint on the ImageNet-100 dataset, we reduce GoogLeNet's latency from 40.32 ms to 34 ms with a 0.14% accuracy reduction on the NVIDIA Jetson Nano. When coupled with quantization, our method can be further improved to only 0.04% drop for GoogLeNet. On the NVIDIA Jetson TX2, we compress VGG-19 from 119.98 ms to 34 ms and even improve its accuracy by 0.5%, and we scale GoogLeNet up from 20.27 ms to 34 ms and achieve higher accuracy by 0.78%. We also open source this framework at https://github.com/ntuliuteam/ZeroBN
