Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of fusing multiple pre-trained speech encoders on the performance of speech-aware large language models (LLMs) for automatic speech recognition (ASR). By exploring various fusion strategies, including learned combinations and Transformer-based architectures, the authors demonstrate that these approaches significantly enhance ASR capabilities across both mono- and multilingual settings, as well as in diarized speech recognition. The findings reveal that such fusion not only boosts performance but does so with minimal computational overhead, making it a practical enhancement for ASR systems.
Fusing multiple speech encoders can dramatically elevate ASR performance without incurring significant computational costs.
Speech-aware large language models (LLMs) can incorporate speech through pre-trained acoustic encoders that project speech features into the LLM embedding space. While the choice of the speech encoder critically influences performance, different encoders often exhibit complementary strengths, motivating their combination. In this work, we investigate whether fusing multiple pre-trained speech encoders can enhance speech-aware LLMs for automatic speech recognition (ASR). We explore several fusion strategies beyond simple feature concatenation, including learned combinations and Transformer-based fusion architectures, and evaluate them across mono- and multilingual ASR settings as well as diarized speech recognition. Our results indicate that carefully fusing multiple parallel speech encoders improves downstream performance in all scenarios with limited computational overhead.