Search papers, labs, and topics across Lattice.
This study explores the tradeoffs in compute allocation for Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER) by analyzing model size, input length, and representation resolution within fixed computational budgets. The findings reveal that while increasing model size initially improves performance, the benefits diminish significantly at larger scales, and an optimal audio duration for SER is identified. Furthermore, the research demonstrates that reducing encoder token resolution can substantially decrease inference costs with minimal impact on performance, providing actionable insights for efficient speech model design.
Diminishing returns on model size reveal that smarter compute allocation can outperform sheer scale in speech processing tasks.
In this paper, we investigate the tradeoffs between compute allocation and model performance for two speech processing tasks: Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). We propose a unified framework that analyzes three fundamental compute dimensions: model size ($x_N$), input length ($x_T$), and representation resolution ($x_V$). Motivated by recent advances in compute optimal scaling for multimodal models, we systematically vary these dimensions to examine their influence on task performance under fixed computational budgets. Our study provides insights into how compute resources can be optimally distributed across model capacity, temporal context, and representational granularity, offering practical guidelines for the design of efficient speech models. Through experiments on LibriSpeech and CREMA-D datasets, we demonstrate non-linear scaling behavior and identify optimal operating points. Our results show that (1) increasing model size yields diminishing returns: scaling Tiny (39M) to Small (244M) reduces WER by 8.22%, whereas Small to Medium (769M) reduces WER by only 2.35%; (2) an optimal audio duration of approximately 4 seconds exists for SER; and (3) reducing encoder token resolution provides an effective mechanism for lowering inference cost, Large-v3 (1540M) with 750 frames requires 2572 GFLOPS whereas with 1500 frames requires 5228 GFLOPS, with less than 3% relative increase in WER. Additionally, LoRA-based adaptation enables efficient finetuning with minimal performance degradation.
