Search papers, labs, and topics across Lattice.
This paper introduces TF-MoE, a sparse Mixture-of-Experts framework designed to improve speech separation efficiency without significantly increasing computational costs. By implementing dynamic expert specialization in both time and frequency dimensions, TF-MoE leverages a mel-band-splitting Conformer backbone to enhance model capacity while maintaining low inference requirements. Experimental results show that TF-MoE outperforms existing models, achieving a +3.8 dB SDR improvement on the Libri2Mix dataset with a comparable inference cost of 4.1 GMACs/s.
TF-MoE achieves a remarkable +3.8 dB improvement in speech separation performance while keeping computational costs low, making it ideal for edge-device applications.
Recent advances in speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.
Microsoft Research