Search papers, labs, and topics across Lattice.
This paper benchmarks and optimizes state-of-the-art ASR architectures, including encoder-decoder, transducer, and LLM-based models, for on-device streaming inference. Through extensive evaluation, NVIDIA's Nemotron Speech Streaming is identified as the most promising architecture. Post-training quantization and operator fusion are then applied to compress the model to 0.67 GB while maintaining high accuracy, achieving 8.20% WER with 0.56s latency on CPU.
You can shrink a state-of-the-art streaming ASR model by 73% with minimal accuracy loss, opening the door for high-quality, low-latency speech recognition on resource-constrained edge devices.
Deploying high-quality automatic speech recognition (ASR) on edge devices requires models that jointly optimize accuracy, latency, and memory footprint while operating entirely on CPU without GPU acceleration. We conduct a systematic empirical study of state-of-the-art ASR architectures, encompassing encoder-decoder, transducer, and LLM-based paradigms, evaluated across batch, chunked, and streaming inference modes. Through a comprehensive benchmark of over 50 configurations spanning OpenAI Whisper, NVIDIA Nemotron, Parakeet TDT, Canary, Conformer Transducer, and Qwen3-ASR, we identify NVIDIA's Nemotron Speech Streaming as the strongest candidate for real-time English streaming on resource-constrained hardware. We then re-implement the complete streaming inference pipeline in ONNX Runtime and conduct a controlled evaluation of multiple post-training quantization strategies, including importance-weighted k-quant, mixed-precision schemes, and round-to-nearest quantization, combined with graph-level operator fusion. These optimizations reduce the model from 2.47 GB to as little as 0.67 GB while maintaining word error rate (WER) within 1% absolute of the full-precision PyTorch baseline. Our recommended configuration, the int4 k-quant variant, achieves 8.20% average streaming WER across eight standard benchmarks, running comfortably faster than real-time on CPU with 0.56 s algorithmic latency, establishing a new quality-efficiency Pareto point for on-device streaming ASR.
