Search papers, labs, and topics across Lattice.
The authors introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to evaluate Speech Large Language Models (SLLMs) under realistic spoken instruction conditions. DOWIS spans 9 tasks and 11 languages, providing 10 prompt variants per task-language pair, across five styles. Benchmarking state-of-the-art SLLMs with DOWIS reveals that text prompts consistently outperform spoken prompts, especially in low-resource and cross-lingual settings, suggesting a need for speech-based prompting in SLLM evaluation.
SLLMs struggle with spoken prompts compared to text, especially in low-resource languages, highlighting a critical gap in current evaluation methodologies.
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.
