Search papers, labs, and topics across Lattice.
PET-DINO is introduced as a universal object detector that supports both text and visual prompts for open-set object detection. The approach uses an Alignment-Friendly Visual Prompt Generation (AFVPG) module to improve text representation alignment and reduce development cycle time. It also employs Intra-Batch Parallel Prompting (IBP) and Dynamic Memory-Driven Prompting (DMD) to enable simultaneous modeling of multiple prompt routes. Experiments show PET-DINO achieves competitive zero-shot object detection performance across various prompt-based detection tasks.
PET-DINO shows that you can get strong zero-shot object detection by unifying text and visual prompts within a single architecture and training strategy.
Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes but faces challenges in aligning text representations with complex visual concepts and the scarcity of image-text pairs for rare categories. This results in suboptimal performance in specialized domains or with complex objects. Recent visual-prompted methods partially address these issues but often involve complex multi-modal designs and multi-stage optimizations, prolonging the development cycle. Additionally, effective training strategies for data-driven OSOD models remain largely unexplored. To address these challenges, we propose PET-DINO, a universal detector supporting both text and visual prompts. Our Alignment-Friendly Visual Prompt Generation (AFVPG) module builds upon an advanced text-prompted detector, addressing the limitations of text representation guidance and reducing the development cycle. We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the iteration level and Dynamic Memory-Driven Prompting (DMD) at the overall training level. These strategies enable simultaneous modeling of multiple prompt routes, facilitating parallel alignment with diverse real-world usage scenarios. Comprehensive experiments demonstrate that PET-DINO exhibits competitive zero-shot object detection capabilities across various prompt-based detection protocols. These strengths can be attributed to inheritance-based philosophy and prompt-enriched training strategies, which play a critical role in building an effective generic object detector. Project page: https://fuweifuvtoo.github.io/pet-dino.
