Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
RepSAM addresses the challenge of adapting foundation models like SAM to robotic vision by identifying and mitigating non-uniform representation shifts across transformer layers. They propose a parameter-efficient fine-tuning (PEFT) framework guided by Centered Kernel Alignment (CKA) to allocate ranks and a multi-modal fusion module to handle complex robotic scenarios. RepSAM achieves 97.9% of full fine-tuning performance with 158x fewer trainable parameters and significantly outperforms other PEFT methods, leading to a 12% improvement in robotic manipulation success rates.
SAM struggles in robotic vision not because of the entire model, but because of representation shifts in shallow layers, and RepSAM exploits this to achieve near-full fine-tuning performance with just 0.6% of the parameters.
Robotic perception in unstructured environments remains challenging despite the zero-shot capabilities of foundation models such as SAM. This work attributes performance degradation to non-uniform representation shifts across transformer layers: shallow layers exhibit substantial domain gaps (CKA < 0.5), whereas deep layers transfer effectively (CKA > 0.7). Based on this observation, we propose RepSAM, a representation-guided parameter-efficient fine-tuning (PEFT) framework for adapting foundation models to robotic vision. RepSAM employs a theoretically grounded CKA-guided rank allocation strategy combined with a multi-modal fusion module for robust handling of challenging robotic scenarios, including transparent objects and cluttered scenes. Experimental evaluation across six benchmarks and robotic manipulation tasks demonstrates that RepSAM achieves 97.9% of full fine-tuning performance (89.0% vs. 90.9% mIoU) while reducing trainable parameters by 158x (from 632M to 4.0M). RepSAM outperforms DoRA by 7.9% mIoU with just 4 hours of training on a single A100 GPU (a 96x reduction from full fine-tuning, which takes 384 GPU-hours). These improvements are statistically significant (p < 0.01) and translate to a 12.0% absolute improvement in robotic manipulation success rates over the LoRA (RGB) baseline.