Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the integration of spatial information into foundation-model-based speaker diarization systems by comparing three distinct approaches: using a beamformer with a single-channel model, employing a multi-channel model, and conditioning the downstream network on explicitly extracted spatial features. The findings reveal that while the beamformer can hinder performance in overlapping speech scenarios, conditioning on spatial features significantly enhances diarization accuracy, outperforming other methods. A detailed error analysis indicates that this conditioning approach effectively mitigates errors associated with using only spectral or spatial features, highlighting its robustness in complex audio environments.
Conditioning on spatial features in diarization systems not only boosts accuracy but also reduces errors that plague traditional methods.
Spatial information gleaned from multi-channel input has been shown to lead to improvements in meeting processing tasks like diarization and source separation. At the same time, diarization based on features extracted by large pretrained single-channel foundation models, such as WavLM, achieved state-of-the-art performance. This work compares three approaches to integrate spatial features into foundation model-based diarization systems: the cascade of a beamformer and a single-channel foundation model, a multi-channel foundation model, and the conditioning of the downstream network on explicitly extracted spatial features. Results show that the beamformer front-end is even detrimental to diarization performance in regions of overlapped speech, while best performance is achieved with the conditioning, demonstrating that the incorporation of explicit spatial features is a competitive approach to foundation-model-supported diarization. This approach is further subjected to a detailed error analysis showing that the conditioning system removes errors to a good extent that would occur when either only spectral or only spatial features were used.