Search papers, labs, and topics across Lattice.
This study introduces Semantic Flip, a novel framework that generates synthetic out-of-distribution (OOD) samples to enhance the refusal capabilities of vision-language models (VLMs) in embodied question answering and spatial localization. By transforming both the user query and the visual memory independently, the framework creates OOD pairs that lack sufficient grounding, enabling a lightweight rejection module to be trained on top of a frozen VLM. The results show that Semantic Flip significantly outperforms existing prompting baselines, achieving an impressive F1 score of 0.9559 on the newly introduced SpaceReject benchmark for spatial localization.
Overconfidence in VLMs can lead to misleading responses, but Semantic Flip effectively trains agents to refuse unanswerable queries with high accuracy.
Detecting unanswerable user queries remains essential for the reliable deployment of real-world embodied agents. However, modern vision-language models (VLMs) often generate overly confident answers even when the available visual memory cannot support the query. Such overconfidence poses various task-dependent risks. The agent may provide misleading information to the user in Embodied Question Answering and select an arbitrary coordinate and physically guide the user there in spatial reasoning for navigation. Despite these high stakes, only a few prior studies directly address when and how an embodied VLM should respond with "I do not know." This work proposes Semantic Flip, a simple yet effective framework that synthesizes auxiliary out-of-distribution (OOD) samples for embodied refusal without requiring external OOD annotations. The key idea is to independently transform the query and video memory to construct auxiliary OOD pairs that lack sufficient visual grounding. These synthesized pairs enable training a lightweight rejection module on top of a frozen pretrained VLM. The module attaches to any existing VLM-based pipeline without retraining the underlying model. Across two complementary benchmarks, Semantic Flip consistently outperforms strong prompting baselines. This work also introduces SpaceReject, a new refusal benchmark for spatial localization with deliberately unanswerable queries over long video memory, where Semantic Flip achieves an $F_1$ score of 0.9559. The source codes and datasets are publicly available at https://github.com/ndb796/SemanticFlip.
