Search papers, labs, and topics across Lattice.
To address the gap between binary refusal benchmarks and real-world prompts, the authors introduce KoNA, a benchmark evaluating vision-language models on selective non-compliance across five failure modes: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. While current VLMs frequently fail to abstain or correct sub-components within compound queries, this diagnostic exposes that models struggle to disentangle valid sub-questions from invalid premises. Fine-tuning on a joint distribution of selective non-compliance and answerable examples substantially improves component-level abstention accuracy while preserving baseline multimodal capabilities.
VLMs that reliably refuse entirely invalid prompts systematically fall apart when unanswerable, infeasible, or unsafe sub-claims are embedded inside otherwise benign compound queries.
Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.
