Search papers, labs, and topics across Lattice.
This paper introduces a hybrid two-stage diffusion transformer architecture designed for instruction-guided audio editing, which effectively balances performance and efficiency through a rectified flow matching approach. By employing a coarse-to-fine strategy, the model first establishes semantic alignment at a low-resolution stage before refining details at a high-resolution stage, enabling precise modifications based on natural language instructions. Experimental results demonstrate significant performance improvements on complex editing tasks while maintaining a compact model that enhances editing efficiency.
Achieving efficient and precise audio editing with a compact model, this hybrid diffusion transformer outperforms existing methods on complex tasks involving overlapping audio events.
Audio editing aims to modify specific content in an existing audio clip according to a natural language instruction while preserving the remaining acoustic content. Despite the remarkable progress of diffusion models, existing training-based editing methods mainly rely on the local inductive biases and cross-attention interaction in convolutional U-Net backbones, which often hinder long-range semantic alignment and precise understanding and localization of instructions. In contrast, diffusion transformers provide stronger global modeling and multimodal fusion, but existing editing architectures usually adopt a simple stack of MMDiT and DiT blocks. Applying joint attention over concatenated audio and text tokens in all blocks results in quadratic complexity with respect to token length. To balance editing performance and efficiency, we propose a hybrid two-stage diffusion transformer architecture for instruction-guided audio editing based on rectified flow matching. It performs joint attention over audio and text tokens to establish coarse semantic alignment at low-resolution stage, then switches to alternating joint-attention and cross-attention blocks to refine editing details at high-resolution stage. This coarse-to-fine strategy enables efficient and accurate instruction-guided audio editing. Experiments show that the proposed framework achieves notable performance gains on challenging editing tasks involving overlapping audio events and complex instructions, while substantially improving editing efficiency with a compact model.
