Search papers, labs, and topics across Lattice.
This paper introduces a few-shot open-vocabulary segmentation (OVS) approach that leverages a support set of pixel-annotated images to overcome limitations of coarse image-level supervision and semantic ambiguity in existing VLMs. They propose a retrieval-augmented test-time adapter that learns a per-image classifier by fusing textual prompts with visual features from the support set. Their learned, per-query fusion strategy achieves stronger synergy between modalities, significantly narrowing the gap between zero-shot and supervised segmentation while maintaining open-vocabulary capabilities.
Just a handful of annotated examples can dramatically close the performance gap between zero-shot and fully supervised open-vocabulary segmentation.
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
