Search papers, labs, and topics across Lattice.
This paper investigates the limitations of softmax-based self-attention in CLIP, which dilutes attention across irrelevant tokens, hindering performance in dense, open-vocabulary prediction tasks. By substituting the traditional softmax with the $\alpha$-entmax transform, the authors effectively zero out low-salience tokens, enhancing the model's focus on relevant contextual cues. The results demonstrate significant improvements in dense semantic segmentation and fine-grained retrieval tasks, particularly when baseline attention is overly diffuse.
Attention sparsification via the $\alpha$-entmax transform can dramatically enhance CLIP's performance in open-vocabulary tasks by filtering out irrelevant context.
Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softmax weights are effective for gathering broad context during pre-training, they spread attention across many low-salience tokens, producing noise that obscures the fine-grained, spatially localized cues required for dense, open-vocabulary prediction. We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers with the $\alpha$-entmax transform, applied across both the standard query-key attention and self-correlation variants. Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser, zeroing contextually irrelevant dependencies while redistributing mass onto the most relevant tokens. We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)-and find the gain from attention sparsification is proportional to how much the baseline attention spreads off the target class.
