Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
NPUsper is a live transcription system designed to enhance the efficiency of Whisper on mobile NPUs by eliminating redundant computations. By detecting hallucinated tokens in real-time through decoder cross-attention patterns, NPUsper minimizes the need for heavy padding and allows for processing of short audio inputs with reduced carryover. The system achieves significant performance improvements, including up to 4.84x lower per-word latency and 88.64% lower average power consumption, while maintaining transcription accuracy comparable to existing methods.
Achieving up to 4.84x lower latency and 88.64% lower power consumption, NPUsper revolutionizes real-time transcription on mobile devices without sacrificing accuracy.
We present NPUsper, a live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation. To avoid the heavy padding used by prior streaming systems, NPUsper detects hallucinated tokens online from temporal patterns in decoder cross-attention, allowing each inference round to process short audio inputs with minimal carryover. For efficient mobile-NPU execution, we propose controlled unrolling, which executes autoregressive decoding as K-step chunk graphs, removing unnecessary KV-cache computation and reducing graph-dispatch overhead. NPUsper achieves up to 4.84x lower per-word latency, up to 33.2x lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy. The code is available at https://github.com/npusper/NPUsper.