Search papers, labs, and topics across Lattice.
This paper investigates the use of deep learning techniques to upsample the covariance matrices from a 4-microphone array to a 32-microphone array, addressing the challenge of limited sensor availability in acoustic imaging. By evaluating five neural network architectures, the authors demonstrate that their models significantly improve spatial resolution and acoustic scene analysis without increasing hardware complexity. The best-performing model achieves a root mean square error (RMSE) of 0.432, showcasing a marked enhancement in sound mapping accuracy compared to the baseline RMSE of 0.548.
Upsampling covariance matrices with deep learning can transform a 4-microphone array into a powerful 32-microphone equivalent, revolutionizing acoustic imaging.
Acoustic imaging visualization is a core methodology in acoustics, enabling spatial analysis of sound sources and acoustic scenes. However, limited sensor availability in practical systems motivate approaches that enhance spatial resolution without increasing the hardware complexity. In this paper, we focus on upsampling virtually a tetrahedral 4-microphone array to a spherical 32-microphone array by estimating the covariance matrices of the channels employing deep learning techniques. Five neural network architectures are investigated for covariance upsampling for acoustic imaging using the real-world STARSS23 dataset. These models are developed to estimate a 32-microphone, time-frequency covariance matrix from a 4-microphone input covariance representation. The proposed architectures are based on 2D convolutional layers to capture the underlying spatial-spectral structure of covariance matrices, and are further enhanced with frequency dynamic convolution to model their frequency-dependent properties. The proposed architectures are evaluated in terms of root mean square error (RMSE) and using delay-and-sum beamforming acoustic imaging. Quantitative results show that all models outperform a random-guess baseline, which yields an RMSE of 0.548, with the best-performing architecture achieving an RMSE of 0.432. We analyze qualitatively the performance of the proposed models through beamforming heatmap visualizations derived from the 4-channel input covariance, the 32-channel ground truth, and the predicted 32-channel covariance matrices. These results demonstrate that covariance upsampling significantly enhances the effective performance of the 4-channel microphone array, producing sound maps that closely resemble those obtained with the 32-channel array.
