Search papers, labs, and topics across Lattice.
This paper introduces Murmur, an efficient inference system for long-form automatic speech recognition (ASR) that balances high accuracy and low latency by optimizing chunk processing. By treating chunk size as a tunable hyperparameter and employing a sliding window KV cache eviction policy, Murmur achieves a 4.2x reduction in latency while maintaining single-pass accuracy on the AMI-IHM dataset. The approach demonstrates that intermediate chunk sizes can effectively manage the trade-off between accuracy and latency, with minimal degradation in performance.
Achieving a 4.2x latency reduction in long-form ASR without sacrificing accuracy could revolutionize real-time speech applications.
Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two. Chunk-based pipelines process audio in parallel windows for low latency, but lose cross-chunk context and need brittle heuristics to align speakers and timestamps at boundaries. Long-context ASR models resolve everything in a single pass for better accuracy, but are an order of magnitude slower. We propose Murmur, an inference system that overcomes this trade-off by operating at two levels. At the inter-chunk level, we revisit the chunk-based pipeline for modern long-context ASR, treating chunk size as a tunable hyperparameter, and show that intermediate chunk sizes strike a good balance of accuracy and latency. At the intra-chunk level, we exploit attention sparsity through a sliding window KV cache eviction policy applied to both output and speech tokens. On AMI-IHM, Murmur matches single-pass accuracy while reducing latency by 4.2x, with further gains from token eviction at less than 1% relative tcpWER degradation. The code of Murmur is available at https://github.com/uw-syfi/Murmur.
UW