Search papers, labs, and topics across Lattice.
This paper introduces Multi-Rank Adaptation (MuRA), a novel framework designed to enhance test-time adaptation for vision-language models by dynamically selecting and fusing adaptation modules based on the visual complexity of inputs. By addressing the limitations of static rank configurations, MuRA improves performance on complex scenes while preventing overfitting on simpler ones, leading to superior accuracy in domain generalization tasks. The method not only achieves state-of-the-art results but also reduces computational and memory overhead, demonstrating its efficiency and effectiveness in real-world applications.
Dynamic adaptation in vision-language models can significantly boost performance while cutting down computational costs.
Vision-language models exhibit remarkable zero-shot capabilities but suffer significant performance degradation under distribution shifts. While test-time adaptation (TTA) via Low-Rank Adaptation offers a parameter-efficient solution, we identify a fundamental bottleneck in current methods: the reliance on static rank configurations. Because visual inputs inherently possess varying information densities, a fixed rank forces an inevitable optimization compromise, leading to underfitting on complex scenes and overfitting on simple ones. To bridge this gap, we propose Multi-Rank Adaptation (MuRA), a novel framework that dynamically selects and fuses adaptation modules of varying capacities based on token-level visual complexity. MuRA synergizes Multi-Rank Orthogonal Decomposition to provide a superior, knowledge-preserving initialization, and Unified Component Fusion with Continuous Router Updating to sustainably learn semantic-to-rank mappings. Furthermore, we provide rigorous theoretical justifications mathematically proving the necessity and gradient stability of this adaptive mechanism. Crucially, MuRA's dynamic design uniquely thrives at the deepest visual layer, capitalizing on the shortest gradient backpropagation path. Extensive experiments demonstrate that MuRA achieves state-of-the-art accuracy across extensive domain generalization and cross-dataset benchmarks while significantly reducing both computational and memory overhead.
Tsinghua AI