Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for certifying the robustness of vision-language models (VLMs) against semantic-level transformations, addressing a critical gap in existing certification methods that primarily focus on geometric or pixel-level changes. By utilizing text prompts as semantic proxies, the framework quantifies the extent of semantic variation while maintaining prediction consistency, allowing for a closed-form characterization of decision boundaries. Experimental results demonstrate its effectiveness in certifying robustness across both synthetic and real-world datasets, highlighting its practical applicability without the need for additional data for each variation.
VLMs can now be certified for robustness against semantic variations without the burden of extra data, transforming how we assess model reliability in real-world applications.
Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.
