Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DETR-ViP, a detection transformer framework designed to improve visual prompted object detection by addressing the lack of global discriminability in visual prompts. They enhance image-text contrastive learning with global prompt integration and visual-textual prompt relation distillation to create more class-distinguishable prompts. Experiments on multiple datasets show DETR-ViP significantly outperforms existing methods in visual prompt detection, demonstrating the effectiveness of their approach.
Visual prompts can now rival text prompts in open-vocabulary object detection, thanks to a novel framework that injects global discriminability.
Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.