Search papers, labs, and topics across Lattice.
This paper introduces AVEX-Prune, a reinforcement learning-based method for dynamic token pruning in audio-visual captioning, which addresses the inefficiencies of existing token retention strategies that often overlook valuable tokens near decision boundaries. By implementing an audio-visual token exchange strategy, AVEX-Prune effectively replaces low-confidence tokens with higher-confidence candidates, optimizing the input to multimodal LLMs without sacrificing output quality. The method achieves a remarkable 40% token retention ratio while maintaining comparable caption generation performance across two prominent models, VILA 1.5-8B and VideoLLaMA 2.
AVEX-Prune retains only the most valuable tokens in audio-visual captioning, achieving 40% retention without sacrificing quality.
Audio-visual captioning generates natural language descriptions from video and audio content. Multimodal LLMs have advanced this task, but both modalities contribute many tokens to the LLM input, where prefill self-attention scales quadratically. Existing token-pruning methods usually retain tokens by attention, saliency, or cross-entropy loss, yet the hard threshold selection makes it difficult to retain tokens that are truly valuable, especially for high-confusing tokens near the decision boundary. To this end, we propose a AVEX-Prune, an RL-based audio-visual dynamic token pruning method in this work. In our AVEX-Prune, an audio-visual token exchange strategy is proposed to select truly valuable tokens by replacing low-confidence retained tokens with high-confidence candidate tokens from the same or the other modality, and measuring the differences in caption generation from token swaps. AVEX-Prune preserves full-token quality at a 40% retention ratio on both VILA 1.5-8B (54.5 vs. 54.6) and VideoLLaMA 2 (57.0 vs. 56.8).
