Search papers, labs, and topics across Lattice.
This paper introduces PCA-Seg, a parallel cost aggregation paradigm for open-vocabulary semantic and part segmentation (OSPS) designed to mitigate knowledge interference between class-level semantics and spatial context in vision-language models. The approach employs an expert-driven perceptual learning (EPL) module with a multi-expert parser and adaptive coefficient mapper to integrate semantic and contextual streams. A feature orthogonalization decoupling (FOD) strategy further reduces redundancy between these streams, leading to state-of-the-art OSPS performance across eight benchmarks with minimal parameter overhead.
By disentangling semantic and contextual cues in vision-language models, PCA-Seg achieves state-of-the-art open-vocabulary segmentation with only 0.35M additional parameters per block.
Recent advances in vision-language models (VLMs) have garnered substantial attention in open-vocabulary semantic and part segmentation (OSPS). However, existing methods extract image-text alignment cues from cost volumes through a serial structure of spatial and class aggregations, leading to knowledge interference between class-level semantics and spatial context. Therefore, this paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the above challenge, enabling the model to capture richer vision-language alignment information from cost volumes. Specifically, we design an expert-driven perceptual learning (EPL) module that efficiently integrates semantic and contextual streams. It incorporates a multi-expert parser to extract complementary features from multiple perspectives. In addition, a coefficient mapper is designed to adaptively learn pixel-specific weights for each feature, enabling the integration of complementary knowledge into a unified and robust feature embedding. Furthermore, we propose a feature orthogonalization decoupling (FOD) strategy to mitigate redundancy between the semantic and contextual streams, which allows the EPL module to learn diverse knowledge from orthogonalized features. Extensive experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
