Search papers, labs, and topics across Lattice.
SEATrack addresses the performance-efficiency trade-off in parameter-efficient multimodal tracking by focusing on cross-modal alignment and efficient global relation modeling. They introduce AMG-LoRA, which combines LoRA with Adaptive Mutual Guidance (AMG) to refine and align attention maps across modalities, mitigating modality-specific biases. Additionally, they propose a Hierarchical Mixture of Experts (HMoE) for efficient global relation modeling in cross-modal fusion. SEATrack achieves state-of-the-art performance with improved efficiency across various multimodal tracking tasks.
Multimodal trackers can achieve state-of-the-art results without ballooning parameter counts by adaptively aligning cross-modal attention maps and using hierarchical mixture of experts for efficient global reasoning.
Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. \href{https://github.com/AutoLab-SAI-SJTU/SEATrack}{\textcolor{cyan}{Code is available}}.
