Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VespaSeg, a modular pipeline for referring expression segmentation that utilizes a compact vision-language model for grounding and MobileSAM for mask generation. The approach achieves a mean intersection over union (mIoU) of 73.64 and processes queries at a rate of 22.8 per second on a standard GPU, demonstrating significant efficiency gains over larger models. Key findings include the effectiveness of ground-truth-box adaptation, which improves segmentation accuracy, and the ability to reduce model complexity without sacrificing performance.
Achieving 73.64 mIoU with a compact model, VespaSeg shows that efficiency in referring expression segmentation doesn't have to come at the cost of accuracy.
Referring expression segmentation requires language conditioned localization and pixel-accurate masks, but monolithic models can be costly to deploy. We present VespaSeg, a modular pipeline that grounds a text query with a compact vision-language model and converts the predicted box to a mask with MobileSAM. We study Florence-2-base, Florence-2-large, and Moondream2 grounders together with targeted adaptation of the grounding and segmentation stages. Under a repository-specific RefCOCO validation protocol containing the first expression for each of 3,811 referenced-object records, the adapted Florence-2-base pipeline obtains 73.64 mean intersection over union (mIoU) and 84.60 precision at IoU 0.5. On an NVIDIA RTX 6000 Ada GPU it processes 22.8 cached-image queries per second with 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for Florence-2-base and 72.82 for Florence-2-large, while the base model is 1.70 times faster and uses 1.17 GB less allocated memory. Ablations show that ground-truth-box adaptation raises MobileSAM mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. These results support compact, modular grounding and segmentation, while also exposing the need for evaluation on the complete standard RefCOCO expression splits and deployment hardware.