Search papers, labs, and topics across Lattice.
This study identifies and formalizes a critical issue in large vision-language models known as Dense Same-Class Attribute Misbinding (DSCAM), where models incorrectly assign attributes to the wrong instances of the same class. The authors introduce InstaBind-Lite, a benchmark designed to measure this misbinding directly, revealing that open-source models exhibit an average Misbinding Rate of 19.84%, while commercial API models show 7.55%. By distinguishing between unsupported generation and recognition failures, the research highlights a significant blind spot in existing evaluation metrics, emphasizing the need for more nuanced assessments of model performance.
Misbinding errors in vision-language models can hide behind high aggregate accuracy, with open-source models misassigning attributes 19.84% of the time.
Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the response as wrong, while object-hallucination metrics may regard both the object and attribute as image-supported; neither reveals the transfer. This study formalizes this blind spot as Dense Same-Class Attribute Misbinding (DSCAM) and presents InstaBind-Lite, a controlled benchmark that makes it directly measurable. Its 524 images contain 529 curated groups of 3-6 same-class entities, 1773 boxed instances, ordered neighbors, distinguishable color-like attributes, and four complementary question levels, yielding 9580 deterministically evaluated questions. Unlike existing protocols, source-instance annotations separate unsupported generation and recognition failure from an attribute copied from another visible entity. Binding-specific metrics further quantify transfer frequency, adjacency, ordinal distance, and intervention effects. Across five open-source and two commercial/API models, the open-source systems average 19.84% Misbinding Rate and the API systems 7.55%; these errors are hidden by aggregate accuracy. Among identifiable transfers, 80.70% and 81.51%, respectively, originate from adjacent instances. Localization and instance-first interventions help selected models but are not universal remedies. InstaBind-Lite therefore turns previously undifferentiated wrong answers into source-identifiable failure categories and tests a reliability dimension that conventional benchmarks cannot determine: whether a model knows not only what is visible, but which instance owns each attribute.
