Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
The authors introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to evaluate Speech Large Language Models (SLLMs) under realistic spoken instruction conditions across 9 tasks and 11 languages. They benchmark state-of-the-art SLLMs using DOWIS, analyzing the impact of prompt modality, style, language, and task type on performance. Their results reveal that text prompts generally outperform spoken prompts, especially in low-resource and cross-lingual scenarios, underscoring the importance of speech-based prompting for comprehensive SLLM evaluation.
Text prompts might be inflating your SLLM's performance: spoken prompts reveal a significant performance gap, especially in low-resource languages.
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.