Search papers, labs, and topics across Lattice.
This paper introduces a computationally efficient guidance method, Selective TFG and Latent-Control Heads (LatCHs), for controlling latent audio diffusion models. LatCHs operate directly in the latent space, bypassing the computationally expensive decoder backpropagation required by existing guidance-based control methods. Experiments on Stable Audio Open demonstrate that LatCHs enable effective control over audio attributes like intensity, pitch, and beats with minimal training resources (7M parameters, 4 hours) while maintaining generation quality.
Steer latent audio diffusion models with minimal overhead using LatCHs, a lightweight guidance approach that avoids costly decoder backpropagation.
Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (\textit{e.g.}, guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and $\approx$ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.
