Search papers, labs, and topics across Lattice.
This paper introduces stride-k subsampling, a method that reduces the number of audio tokens processed by the Whisper model by retaining every k-th token, specifically demonstrating that k=2 maintains baseline word error rates (WER) while significantly decreasing computational load. The approach leverages the inherent redundancy in Whisper's fixed token representation, achieving a 75% reduction in audio tokens and a 52-58% decrease in GFLOPs with minimal WER impact on most benchmarks. Additionally, this method is applicable to Whisper-based speech language models, resulting in reduced end-to-end latency without the need for retraining or additional computation.
Reducing audio tokens by 75% with stride-k subsampling preserves performance while slashing computational costs鈥攏o retraining required.
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
