Search papers, labs, and topics across Lattice.
This paper introduces a universal speech enhancement model capable of adapting to various real-time applications by controlling both algorithmic and computational latency. The model employs configurable look-ahead frames and an early-exit mechanism to optimize performance across different latency budgets, while a two-stage training strategy enhances its efficiency. The results show that this single model can match the performance of specialized models without the need for retraining, streamlining deployment in diverse scenarios.
A single universal speech enhancement model can effectively adapt to multiple latency requirements without sacrificing performance, challenging the need for specialized models.
Different real-time speech applications impose distinct latency budgets, often requiring separately trained enhancement models for each scenario. In this paper, we propose a one-for-all, real-time universal speech enhancement model that provides explicit control over both algorithmic and computational latency. Algorithmic latency is flexibly adjusted via configurable look-ahead frames. To avoid learning inefficiency caused by varying padding configurations, we introduce parallel convolutional layers corresponding to different look-ahead settings. Computational latency is controlled through an early-exit mechanism, enabling inference at different network depths. To narrow the performance gap between specialized and flexible models, we propose a two-stage training strategy with a shared-to-multiple decoder transition. Overall, the proposed framework enables a single model to be deployed across diverse latency budgets without retraining separate models. Model weights are available for download at: https://huggingface.co/nvidia/Real-time_RE-USE
