Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Semantic-Agnostic aNd Shape-Aware (SANSA) segmentation, a new paradigm for vision-language segmentation that uses only non-semantic textual descriptions to focus on intrinsic visual properties. They propose two strategies for generating SANSA prompts: dictionary-constrained and example-guided, which are then used to finetune segmentation models. Finetuning with SANSA prompts improves mIoU by up to 20% on the new task, while preserving performance on standard semantic prompts, demonstrating the value of low- and mid-level visual reasoning.
Vision-language models can get a 20% segmentation boost by learning to parse *non-semantic* textual descriptions of shape and texture.
Vision-language segmentation models have recently achieved strong performance by leveraging high-level semantic object categories expressed in natural language. However, this semantic dependence limits their ability to reason about intrinsic visual properties such as shape, geometry, or texture, which are essential in many real-world applications. In this work, we introduce Semantic-Agnostic aNd Shape-Aware (SANSA) segmentation, a new paradigm that requires segmentation models to operate solely from non-semantic textual descriptions. To this end, we propose two strategies to generate SANSA segmentation prompts based on either dictionary constraints or example guidance, both generating semantic-agnostic textual descriptions. These prompts are then used to finetune segmentation models under semantic-agnostic supervision. Experiments show that finetuning on SANSA prompts yields up to a 20% mIoU improvement on this new segmentation task, compared to pretrained state-of-the-art models, while maintaining strong performance on standard semantic prompts. These results highlight the importance of low- and mid-level visual reasoning for improving the generalization and controllability of vision-language segmentation models.