Search papers, labs, and topics across Lattice.
This paper introduces instruction-based vector steering, a novel method that contrasts activations from differently instructed prompts to redirect temporal attention in Large Audio-Language Models (LALMs) while keeping the audio input fixed. The approach significantly enhances the model's focus on acoustically relevant regions of audio signals, achieving a notable improvement in the recovery of queried sound events compared to standard prompting and random baselines. The findings reveal a mechanistic property of LALMs, demonstrating that this steering technique can effectively probe the latent temporal structure encoded within these models without requiring additional training.
Instruction-based steering can redirect attention in LALMs to acoustically relevant regions, achieving over 60% overlap with ground-truth sound event locations without any training.
Large Audio-Language Models (LALMs) excel at audio understanding but expose little about where in an audio signal they attend. We introduce instruction-based vector steering, which constructs a steering vector by contrasting activations from differently instructed prompts while keeping the audio fixed. Through a systematic probe of LALM attention, we find that - unlike standard prompting or audio-based steering - this intervention significantly redistributes the temporal attention allocated to audio tokens, concentrating it on acoustically relevant regions. We then show that this attention shift is behaviorally meaningful: in a controlled three-event setting, reading out the temporal position of maximal steering-induced attention change recovers the location of a queried sound event without any training, attaining 60.87% and 68.72% overlap with ground-truth intervals on Qwen2-Audio and Audio Flamingo 3, far above direct prompting (31.84%, 46.75%) and random baselines (27.74%). Our results characterize a mechanistic property of instruction-based steering in LALMs and provide a training-free probe for the latent temporal structure these models encode.
