Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel beamforming pipeline that enhances speech extraction from moving speakers using only an initial direction estimate, addressing the limitations of deep neural network parameterization in dynamic environments. By decoupling neural temporal-spectral processing from linear spatial processing, the method achieves robust and generalizable performance across various ambisonics orders. Evaluation on both synthetic and real-world data demonstrates the approach's effectiveness in challenging scenarios, including closely spaced and crossing speakers in dynamic settings.
Achieving robust speech enhancement in dynamic environments with only an initial direction estimate could revolutionize how we approach multi-speaker scenarios in audio processing.
Linear spatial filters (beamformers) enable robust, generalizable and interpretable speech enhancement with performance guarantees under ideal parameterization. Modern beamformers are often parameterized by deep neural networks, whose performance degrades in dynamic scenarios with multiple moving speakers of unknown directions. We propose a data-driven beamforming pipeline, which only requires an estimate of the target's initial direction. Building on a higher-order ambisonics representation, we show that neural temporal-spectral processing can be decoupled from linear spatial processing, and thereby achieve generalizable and array-agnostic enhancement. By incorporating autoregression into a frame-wise causal framework, we maintain consistent performance throughout fast speaker motion and long recordings. Evaluation on synthetic data demonstrates robust enhancement under challenging conditions with closely spaced and crossing speakers. Real-world recordings in a dynamic office meeting scenario complement these findings and show generalizability across varying ambisonics orders.