Search papers, labs, and topics across Lattice.
This paper addresses the limitations of the InfoNCE objective in dense-caption retrieval, which can prematurely saturate due to the presence of near-duplicate captions. The authors introduce HN-CLIP, a novel approach that leverages the text encoder's geometry to create adaptive similarity margins for negatives, enhancing the contrastive learning process without the need for additional data or complex preprocessing. Experimental results demonstrate that HN-CLIP outperforms existing methods by 2.4-4.3 R@1 while significantly accelerating training speed, achieving superior performance even with reduced training data.
Adaptive similarity margins in HN-CLIP boost retrieval accuracy by up to 4.3% while training 2.4x faster than leading methods.
Dense-caption retrieval has recently been improved by introducing segmentation, edge maps, LLM-filtered captions, and cross-modal modules into contrastive fine-tuning. However, these methods largely inherit the same InfoNCE objective, whose optimization can prematurely saturate under a strong pre-trained initialization: on dense captions, the loss falls below 10^{-3} on 80% of batches within the first epoch, while its gradient reaches exact zero in fp32 in 47% of measurements. We find that this behavior is closely related to the large number of near-duplicate captions in dense-caption benchmarks, where a few highly similar negatives remain unresolved after the easy majority has already been separated. As a remedy, we introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins. Specifically, a detached caption-similarity matrix is added to the negative logits, assigning larger margins to more similar captions without mining, synthesizing, or resampling negatives. The resulting objective requires only one caption-similarity matrix and a masked logit addition during training, with no auxiliary data, additional parameters, offline preprocessing, or inference-time overhead. Extensive experiments on four dense-caption retrieval benchmarks show that HN-CLIP improves over the strongest competitors by +2.4--+4.3 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP. Moreover, the proposed objective improves all six tested fine-tuning frameworks on the in-domain benchmarks and reaches the strongest full-data baseline with only 20% of the training data.
