Search papers, labs, and topics across Lattice.
VibeVoice-ASR-BitNet is a highly optimized, compressed version of VibeVoice-ASR designed for real-time inference on edge CPUs, utilizing heterogeneous quantization techniques to enhance performance. The model employs full-pipeline INT8 quantization for the VAE acoustic tokenizer and BitNet-style ternary weights for the autoregressive language model, achieving significant speed improvements. With a real-time factor (RTF) of less than 1 and a speed increase of 1.6-2.3 times over Whisper.cpp, it maintains competitive accuracy levels despite aggressive compression strategies.
Achieving real-time ASR performance on edge devices, VibeVoice-ASR-BitNet outpaces Whisper.cpp by up to 2.3x while only modestly sacrificing accuracy.
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF<1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.
