Search papers, labs, and topics across Lattice.
This paper identifies and analyzes "center bias" in CLIP models, where the model disproportionately focuses on the central region of an image and overlooks objects near the boundaries. Through embedding decomposition and attention map analysis, the authors find that information loss during visual embedding aggregation, particularly due to pooling, causes relevant off-center concepts to vanish. They demonstrate that this bias can be mitigated using training-free visual prompting and attention redistribution techniques.
CLIP models suffer from a surprisingly strong "center bias," causing them to miss important objects outside the image's central region, even when those objects are crucial for accurate vision-language understanding.
Recent research has shown that contrastive vision-language models such as CLIP often lack fine-grained understanding of visual content. While a growing body of work has sought to address this limitation, we identify a distinct failure mode in the CLIP family, which we term center bias, that persists even in recent model variants. Specifically, CLIP tends to disproportionately focus on the central region of an image, overlooking important objects located near the boundaries. This limitation is fundamental as failure to recognize relevant objects makes it difficult to perform any sophisticated tasks that depend on those objects. To understand the underlying causes of the limitation, we conduct analyses from both representation and attention perspectives. Using interpretability methods, i.e., embedding decomposition and attention map analysis, we find that relevant concepts especially those associated with off-center objects vanish from the model's embedding in the final representation due to information loss during the aggregation of visual embeddings, particularly the reliance on pooling mechanisms. Finally, we show that this bias can be alleviated with training-free strategies such as visual prompting and attention redistribution by redirecting models' attention to off-center regions.
