Search papers, labs, and topics across Lattice.
This study investigates various approaches to improve speech recognition systems in complex "cocktail party" scenarios, where multiple speakers are present. By analyzing systems tested on the CHiME-9 MCoRec task, the research identifies three effective strategies: audio-visual target speech separation, enhanced audio-visual speech recognition, and the integration of large language models for speaker grouping. The findings reveal that the best-performing system achieves a 57% relative error reduction, challenging the notion that high speech overlap is the main difficulty in these environments.
High speech overlap isn't the primary challenge in cocktail-party scenarios; innovative audio-visual strategies and large language models can cut recognition errors by 57%.
Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.
CMU ML