Search papers, labs, and topics across Lattice.
This paper introduces AudioWeave, a unified model for text-to-audio generation and audio editing that eliminates the need for task-specific architectures. By employing a joint condition modeling approach with factorized position embeddings and a progressive multistage training strategy, the model effectively mitigates task competition and catastrophic forgetting. Experimental results demonstrate that AudioWeave achieves competitive performance with traditional task-specific models while simplifying the overall system architecture.
AudioWeave challenges the status quo by achieving competitive audio generation and editing performance without the complexity of task-specific architectures.
With the growing focus on audio in multimedia applications, numerous advanced works on audio generation have emerged. Existing studies typically treat text-to-audio (TTA) and other related audio generation tasks, such as instruction-based audio editing, as independent challenges, adopting task-specific architectures or modules. This absence of a unified modeling paradigm substantially increases the overhead and complexity of building a system for both audio generation and editing, while also leading to limited scalability. To address this issue, we introduce AudioWeave, a unified model for TTA and audio editing without additional task-specific components. Specifically, we propose a joint condition modeling approach with a factorized position embedding, enabling the diffusion transformer backbone to operate under heterogeneous inputs of TTA and audio editing. We further propose a progressive multistage training strategy to mitigate task competition and catastrophic forgetting caused by interference among multiple tasks. This in turn helps maintain the performance of each individual task and may even lead to improvements in certain aspects. Experimental results on TTA task and six audio editing tasks show that our unified model achieves competitive performance with task-specific models, laying a groundwork for further exploration of unified audio generation models.
