Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Prototype-Guided Text Calibration (PTC), a method for training-free open-vocabulary semantic segmentation (OVSS) that addresses the semantic gap between generic text embeddings and instance-specific visual representations. By selecting reliable visual evidence to create category-specific prototypes and calibrating text embeddings accordingly, PTC enhances the alignment of visual and textual information without requiring additional training. Extensive experiments demonstrate that PTC improves segmentation accuracy and completeness across multiple benchmarks, validating its effectiveness as a plug-and-play enhancement for existing OVSS methods.
Calibrating text embeddings with visual prototypes boosts segmentation accuracy without any additional training, bridging the gap between generic concepts and specific visual instances.
Training-free open-vocabulary semantic segmentation (OVSS) partitions an image into semantically distinct regions based on arbitrary text descriptions, without learning any additional parameters. However, existing methods typically focus on improving visual representations while treating text embeddings that encode only generic category concepts as fixed classification references. The resulting semantic gap between these generic concepts and the visual representations that capture the specific appearances of target instances often causes incomplete masks and erroneous predictions in non-target regions. Inspired by the symbol-percept correspondence underlying perceptual anchoring, we propose Prototype-Guided Text Calibration (PTC) for training-free OVSS. In the Perceiving stage, PTC selects reliable visual evidence based on initial matching scores to construct category-specific visual prototypes. In the Anchoring stage, PTC uses these prototypes to calibrate their corresponding text embeddings, with the calibration strength adaptively adjusted based on the amount of visual evidence. Consequently, the calibrated text embeddings align more accurately with instance-specific visual representations while preserving generic category semantics and open-vocabulary generalization. Moreover, PTC requires neither additional training nor external models and can serve as a plug-and-play module for existing methods. Extensive experiments across eight benchmarks show that PTC significantly enhances the performance of six representative methods and yields more complete and accurate segmentation results. These results validate PTC as a simple and effective approach to improving visual-text alignment.