Search papers, labs, and topics across Lattice.
This paper introduces U-PAST, a hybrid transformer-U-Net architecture designed for single-channel speech enhancement that leverages self-attention to model long-range time-frequency dependencies in complex spectrograms. Evaluated across multiple datasets, U-PAST outperforms existing models under acoustic mismatch conditions and closely approaches the performance of larger convolutional baselines while maintaining a smaller parameter footprint. The architecture's ability to achieve superior perceptual quality under dataset mismatch conditions highlights its effectiveness in real-world applications.
U-PAST achieves state-of-the-art speech enhancement performance with a fraction of the parameters, outperforming larger models in challenging acoustic conditions.
Convolutional neural networks (CNNs), used widely and successfully in audio enhancement, capture long-range time-frequency dependencies only indirectly, through successive convolution and pooling. Here, we present U-PAST, a hybrid transformer-U-Net architecture that addresses this limitation through self-attention dependency-modeling in the complex spectrogram domain. U-PAST tokenizes a complex STFT representation, similarly to the magnitude spectrogram tokenization of the Audio Spectrogram Transformer (AST), applies a multi-layer transformer encoder, and reconstructs the enhanced complex spectrogram with a U-Net-style decoder. We evaluate four architectural variants with between 1.17M and 2.40M parameters on the DNS Challenge, VoiceBank-DEMAND, and LibriMix corpora under matched, acoustic mismatch, and two-dataset mismatch conditions. U-PAST attains the best SI-SDR of any evaluated model under acoustic mismatch and closely trails substantially larger convolutional and time-domain baselines by 0.26 dB to 0.63 dB SI-SDR under the remaining three conditions while achieving the strongest perceptual (DNSMOS) quality under dataset mismatch. The largest evaluated configuration, U-PAST-H (2.40M parameters), is consistently the strongest variant of the family, offering an attractive performance-to-cost trade-off at a small parameter footprint.
