Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates the internal representations of semantic musical concepts within audio diffusion models using activation patching. It finds that a small subset of attention layers controls high-level concepts like instruments, vocals, and genre. By applying Contrastive Activation Addition and Sparse Autoencoders to these layers, the authors achieve precise control over generated audio, enabling targeted modifications of musical elements like tempo and mood.
Unlock precise control over audio diffusion models by steering activations in a small, shared subset of attention layers responsible for high-level musical concepts.
Audio diffusion models can synthesize high-fidelity music from text, yet their internal mechanisms for representing high-level concepts remain poorly understood. In this work, we use activation patching to demonstrate that distinct semantic musical concepts, such as the presence of specific instruments, vocals, or genre characteristics, are controlled by a small, shared subset of attention layers in state-of-the-art audio diffusion architectures. Next, we demonstrate that applying Contrastive Activation Addition and Sparse Autoencoders in these layers enables more precise control over the generated audio, indicating a direct benefit of the specialization phenomenon. By steering activations of the identified layers, we can alter specific musical elements with high precision, such as modulating tempo or changing a track's mood.