Search papers, labs, and topics across Lattice.
This study investigates the internal pathways of Audio-Visual Large Language Models (AVLLMs) to understand how audio and visual signals influence predictions. By analyzing two input configurations鈥攁udio-visual video and multiple interleaved audio-visual items鈥攖he research reveals that AVLLMs utilize a sequential information flow for video inputs, while shifting to parallel streams for interleaved items. Notably, the findings indicate that certain audio-visual tokens can be discarded after their information is integrated, leading to minimal impact on predictions and even slight improvements, thus enhancing inference efficiency across various tasks and models.
AVLLMs can efficiently discard certain audio-visual tokens without sacrificing prediction quality, revealing a surprising flexibility in multimodal processing.
Multimodal Large Language Models (MLLMs) can listen and see, but how do audio and visual signals actually travel through the network to shape an answer? Despite their growing role in research and real-world applications, the internal pathways through which audio and visual tokens influence the final prediction remain poorly understood. In this study, we examine audio-visual information flow inside Audio-Visual Large Language Models (AVLLMs), tracing how AVLLMs route, utilize, and integrate audio and visual information across two input configurations, audio-visual video and multiple interleaved audio-visual items. We find that for audio-visual video, AVLLMs follow the sequential information flow pathway established for VLMs and VideoLLMs, with audio and visual contribution flowing along this pathway in proportion to the task's reliance on each modality. In settings with multiple interleaved audio-visual items, this routing shifts to different parallel streams. Furthermore, we demonstrate that audio-visual and other token types can be discarded once their information is transferred to LLM, with minimal impact on the model's prediction or even slight improvement, generalizing across multiple tasks and datasets, enabling more efficient inference. These findings hold across multiple models and scales, Qwen2.5-Omni and Video-SALMONN2 Plus at 3B and 7B scales, leading to hypotheses on why these flow structures emerge. Together, these results deliver the first coherent picture of how AVLLMs orchestrate sound and sight inside the network and lay the groundwork for the next wave of interpretability, design, and efficiency advances in audio-visual and broader MLLMs.
