Search papers, labs, and topics across Lattice.
This paper introduces S4ECA, a novel framework for Referring Remote Sensing Image Segmentation (RRSIS) that enhances cross-modal alignment by utilizing a dual-encoder adapter architecture. By implementing a parameter-efficient adaptation strategy, the model updates only 2.4% of the backbone parameters while effectively capturing complex dependencies between language and aerial imagery. The results show that S4ECA achieves state-of-the-art performance on benchmark datasets, significantly improving efficiency and precision in segmenting remote sensing images based on natural language queries.
Achieving state-of-the-art RRSIS performance with just 2.4% parameter updates challenges the norm of full fine-tuning in multimodal models.
Referring Remote Sensing Image Segmentation (RRSIS) seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image. While existing RRSIS models have benefited from large-scale foundation models, they predominantly rely on full fine-tuning. These approaches are computationally intensive and may weaken the generalization ability of pre-trained models, as extensive fine-tuning on significantly smaller downstream datasets can distort the well-structured feature representations learned during large-scale pre-training. Although Parameter-Efficient Tuning (PET) offers a potential alternative, existing PET frameworks primarily focus on single-modal optimization, failing to capture the complex cross-modal dependencies required for multimodal reasoning, while simultaneously struggling to bridge the substantial domain gap between natural scenes and aerial imagery. To address these limitations, we propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment (S4ECA), which enables effective and efficient cross-modal interaction through parameter-efficient adaptation. Specifically, we design a dual-encoder adapter architecture. The textual adapter employs learnable queries to distill highly semantic language proxies from word-level embeddings, facilitating early grounding. Simultaneously, the visual adapter refines hierarchical feature representations through a multi-scale dense extractor, followed by a language-guided scale and spatial selection mechanism that dynamically emphasizes relevant visual contexts, ensuring precise cross-modal alignment. By updating only 2.4% of the backbone parameters, our proposed model achieves state-of-the-art performance on the RRSIS-D and RefSegRS datasets, demonstrating superior efficiency and precision in complex aerial scenarios.
