Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically benchmarks four inference strategies—plain PyTorch, ONNX Runtime, OpenVINO, and TensorRT—specifically for industrial machine vision applications under real-world deployment conditions. The evaluation, which spans various CPU and GPU hardware platforms and includes both CNNs and a transformer-based model, reveals that OpenVINO excels in CPU inference times while TensorRT leads on GPUs, although it falls short against plain PyTorch for the transformer model. These findings provide critical insights for practitioners seeking optimal performance in edge deployment scenarios where latency and efficiency are paramount.
OpenVINO outperforms all competitors for CPU inference in industrial machine vision, but TensorRT's supremacy on GPUs doesn't extend to transformer models.
Edge deployment is often the preferred solution for industrial machine vision systems when low latency, data security, or limited connectivity are critical requirements. Several frameworks are available to optimise inference on edge devices; however, relatively few studies have systematically compared their inference-time performance under industrial deployment conditions. In this work, we present a comparative study of four widely used approaches for machine vision inference in industrial settings: plain PyTorch, ONNX Runtime, OpenVINO, and TensorRT. The evaluation focuses on inference time, covers several CPU- and GPU-based hardware platforms, and includes both conventional convolutional neural networks and a transformer-based vision model. For the hardware platforms and models evaluated, the results show that OpenVINO achieves the lowest inference time on CPUs, while TensorRT achieves the lowest inference time on GPUs. However, TensorRT does not outperform plain PyTorch for the transformer-based model considered in this study.