Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a duration-aware scheduling approach for Automatic Speech Recognition (ASR) serving pipelines, addressing the limitations of traditional first-come-first-served (FCFS) methods that lead to head-of-line blocking. By leveraging audio duration as a proxy for job processing time, the authors integrate Shortest Job First (SJF) and Highest Response Ratio Next (HRRN) algorithms into the vLLM framework. The results show that while SJF can significantly reduce median end-to-end latency by up to 73% at high loads, HRRN provides a balanced solution, achieving a 28% reduction in median latency with a maximum tail-latency increase of 24%, even under workload drift.
Duration-aware scheduling can cut ASR latency by up to 73%, but balancing this with tail latency is crucial for practical deployment.
Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to $73\%$ at high load, but increases $90$th-percentile tail latency by up to $97\%$ due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to $28\%$ while bounding tail-latency degradation to at most $24\%$. These gains persist under workload drift, with no throughput penalty and $<0.1$\,ms scheduling overhead per request.