Search papers, labs, and topics across Lattice.
To evaluate multimodal speech models in specialized domains, the authors establish $S^3$-Bench, a benchmark spanning 10 scientific disciplines across single-turn speech QA and simulated multi-turn agent dialogues. Testing models on scientific voice interaction exposes unique failure modes that general audio benchmarks miss, including the verbalization of symbolic expressions, rare technical nomenclature, and spoken abbreviations. By decomposing conversational turns into perception, reasoning, and pronunciation stages, the evaluation reveals severe bottlenecks in symbolic verbalization and persistent degradation in multi-turn user adaptation.
Even top-performing voice MLLMs break down when forced to talk science, routinely failing to verbalize symbolic notation, decode technical jargon, and adapt across progressive multi-turn interactions.
The advance of multimodal large language models (MLLMs) has fundamentally reshaped the paradigm of human-computer interaction, especially speech interaction models capable of seamless conversations. Despite remarkable performance as general voice assistants, their performance in specialized domains remains underexplored, particularly in scientific areas. Scientific interactions introduce formidable challenges, involving rare technical terminology, spoken norms of abbreviations, and the natural verbalization of symbolic special expressions. In this paper, we introduce S$^3$-Bench, a systematic evaluation framework covering 10 major disciplines, consisting of a Knowledge set for speech question-answering and a Dialogue set for multi-turn progressive interactions with simulated user agents. By decomposing a complete atomic turn into stages of speech recognition, perception, knowledge utilization with reasoning, and response pronunciation, we systematically characterize the common challenges and performance tradeoffs of existing approaches. Furthermore, experiments on multi-turn interactions reveal persistent limitations in user adaptation and the generation of accurate, comprehensive, and efficient responses.
