Search papers, labs, and topics across Lattice.
This paper introduces TALENT, a parameter-efficient tuning framework for referring image segmentation that tackles the "non-target activation" (NTA) issue where visual features activate unrelated objects of the same category. TALENT employs a Rectified Cost Aggregator (RCA) for efficient feature aggregation and a Target-aware Learning Mechanism (TLM) comprising contextual pairwise consistency learning and target-centric contrastive learning to calibrate NTA. Experiments demonstrate that TALENT achieves state-of-the-art performance, improving mIoU by 2.5% on the G-Ref validation set.
Existing methods for referring image segmentation struggle to focus on the target object, but TALENT solves this with a novel learning mechanism that dramatically improves segmentation accuracy.
Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the `non-target activation'(NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrate `NTA'into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA'effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5\% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.
