Search papers, labs, and topics across Lattice.
This paper introduces a stage-adaptive training strategy for audio diffusion models that dynamically adjusts the optimization process based on a progress-based regime variable derived from SSL-space discrepancy. The approach incorporates decayed SSL guidance for early semantic bootstrapping, self-adaptive timestep sampling, and structure-aware regularization. Experiments on text-conditioned audio generation and audio-conditioned super-resolution demonstrate improved convergence and performance compared to static baselines, highlighting the benefits of stage-dependent optimization.
Audio diffusion models can be trained more efficiently by dynamically adjusting optimization strategies based on the evolving balance between semantic acquisition and fine-detail refinement during training.
Recent progress in diffusion-based audio generation and restoration has substantially improved performance across heterogeneous conditioning regimes, including text-conditioned audio generation and audio-conditioned super-resolution. However, training audio diffusion models remains computationally expensive, and most existing pipelines still rely on static optimization recipes that treat the relative importance of training signals as fixed throughout learning. In this work, we argue that a major source of inefficiency lies in the evolving balance between semantic acquisition and generation-oriented refinement. Early training places stronger emphasis on acquiring condition-aligned semantic structure and coarse global organization, whereas later training increasingly emphasizes temporal consistency, perceptual fidelity, and fine-detail refinement. To characterize this evolving balance, we introduce a progress-based regime variable derived from the training-time slope of an SSL-space discrepancy, which measures semantic progress during training. Based on this signal, we develop three complementary stage-aware mechanisms: decayed SSL guidance for early semantic bootstrapping, self-adaptive timestep sampling driven by the regime variable, and structure-aware regularization activated from convergent grouped organization in parameter space. We evaluate these mechanisms on text-conditioned audio generation and audio-conditioned super-resolution. Across both settings, the proposed stage-aware strategies improve convergence behavior and yield gains on the primary generation and spectral reconstruction metrics over standard static baselines. These results support the view that efficient audio diffusion training can benefit from treating external guidance, internal organization, and optimization emphasis as stage-dependent components rather than fixed ingredients.
