Search papers, labs, and topics across Lattice.
This paper introduces Visual Concept Inference from Sets (VICIS), a novel task that assesses vision-language models' ability to infer shared concepts from example image sets and apply them to new queries. The authors demonstrate that existing state-of-the-art VLMs struggle with this task, often neglecting visual context and producing biased outputs. By proposing a new training framework and architecture, the study shows significant improvements in generating accurate and diverse outputs, with the model effectively generalizing to unseen concepts and modalities, such as sketches.
State-of-the-art vision-language models fail to leverage visual context, leading to biased outputs, but a new training framework shows they can learn to infer concepts from image sets effectively.
Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.
