Search papers, labs, and topics across Lattice.
To resolve the poor visual grounding of open-vocabulary detectors on functional or ambiguous language, the authors develop Vague2Detect, a pipeline that decouples semantic intent resolution from image-level object localization. Bridging this gap is essential for embodied agents that must interpret high-level task instructions rather than explicit canonical category labels. On a benchmark combining custom household imagery and Open Images V7, the framework boosts YOLO-World's baseline Vague Prompt Success Rate from 32% to 61% via knowledge-base retrieval, and up to 85% when backed by dynamic LLM candidate generation.
Standard open-vocabulary detectors fail on nearly 70% of ambiguous, task-driven queries, but layering structured semantic retrieval and dynamic LLM concept expansion over frozen detectors pushes grounding success to 85%.
Real-world detectors must often interpret functional or ambiguous prompts, yet conventional models such as YOLO remain restricted to fixed class lists. Even open-vocabulary models like YOLO-World frequently misalign vague language with the intended objects. Building on our prior work Commonsense-Guided Open-World Object Detection Using LLMs and Visual-Semantic Matching, we address YOLO-World's limitations in grounding task-driven queries. We propose Vague2Detect, a hybrid pipeline in which a fine-tuned Sentence-BERT retrieves candidates from a structured household Knowledge Base (KB), and YOLO-World verifies their presence in the image. For prompts outside the KB, a large language model (GPT-3.5-turbo) generates candidate descriptions, dynamically expanding the KB to cover novel concepts. On a benchmark of household scenes using custom images and an Open Images V7 subset, YOLO-World alone achieves only 32% Vague Prompt Success Rate (VPSR), the ability to map ambiguous queries to correct detections. In contrast, Vague2Detect improves performance to 61% VPSR with high precision, and up to 85% when augmented with GPT fallback.
