Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents the USTC-NERCSLIP team's system for the CHiME-9 MCoRec Challenge, which addresses the difficult task of recognizing and clustering multiple concurrent conversations in indoor environments with high speech overlap. The system employs a multimodal cascaded approach, integrating per-speaker visual streams from 360° video with single-channel audio, and enhances Active Speaker Detection (ASD), Audio-Visual Target Speech Extraction (AVTSE), and Audio-Visual Speech Recognition (AVSR) modules using improved audio-visual pretrained models. By incorporating Whisper and LLM techniques into the AVSR module and applying ROVER for system fusion, the system achieves a Speaker WER of 31.40% and a Joint ASR-Clustering Error Rate (JACER) of 15.70%, with LLM-based zero-shot conversational clustering achieving a speaker clustering F1 score of 1.0.
LLMs can perfectly cluster speakers in overlapping multi-party conversations, enabling near-perfect Joint ASR-Clustering Error Rate in challenging CHiME-9 tasks.
This report details our submission to the CHiME-9 MCoRec Challenge on recognizing and clustering multiple concurrent natural conversations within indoor social settings. Unlike conventional meetings centered on a single shared topic, this scenario contains multiple parallel dialogues--up to eight speakers across up to four simultaneous conversations--with a speech overlap rate exceeding 90%. To tackle this, we propose a multimodal cascaded system that leverages per-speaker visual streams extracted from synchronized 360 degree video together with single-channel audio. Our system improves three components of the pipeline by leveraging enhanced audio-visual pretrained models: Active Speaker Detection (ASD), Audio-Visual Target Speech Extraction (AVTSE), and Audio-Visual Speech Recognition (AVSR). The AVSR module further incorporates Whisper and LLM techniques to boost transcription accuracy. Our best single cascaded system achieves a Speaker Word Error Rate (WER) of 32.44% on the development set. By further applying ROVER to fuse outputs from diverse front-end and back-end variants, we reduce Speaker WER to 31.40%. Notably, our LLM-based zero-shot conversational clustering achieves a speaker clustering F1 score of 1.0, yielding a final Joint ASR-Clustering Error Rate (JACER) of 15.70%.