Search papers, labs, and topics across Lattice.
This paper explores the mechanistic interpretability of the Multitrack Music Transformer (MMT) by introducing a framework for deterministic attribute modulation through inference-time activation steering, specifically targeting Pitch and Duration. By employing the Difference-in-Means (DiffMean) methodology, the authors successfully isolate latent directions for these signal attributes and validate the Linear Representation Hypothesis, showing a high correlation between steering magnitude and attribute shift. The proposed Dual Steering framework, which utilizes Gram-Schmidt Orthogonalization, effectively mitigates feature entanglement, allowing for independent control of attributes while maintaining signal integrity against autoregressive conditioning.
Independent control over musical attributes like pitch and duration is now achievable without retraining, thanks to a novel activation steering framework.
Transformer-based architectures have significantly advanced the generation of complex symbolic sequences, yet a significant gap remains in achieving fine-grained, interpretable control over discrete signal attributes. This paper investigates the mechanistic interpretability of the Multitrack Music Transformer (MMT) and proposes a framework for deterministic attribute modulation without retraining to bridge this gap via inference-time activation steering. Utilizing the Difference-in-Means (DiffMean) methodology, we isolate latent directions for signal attributes, specifically Pitch and Duration, within the residual stream. We validate the Linear Representation Hypothesis in this domain, achieving high correlation between steering magnitude and attribute shift. To address the inherent feature entanglement in multi-attribute steering, we introduce a Dual Steering framework utilizing Gram-Schmidt Orthogonalization. Experimental results demonstrate that this geometric decoupling reduces conceptual interference and signal degradation compared to naive vector addition, enabling independent deterministic control even against strong autoregressive conditioning.
