Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper introduces RePair, a novel method that transforms retrieval failures into useful counterfactual hard pairs for vision-language tasks. By leveraging the semantics of top-ranked false positives and minimally correcting their residuals, RePair generates hard positive and negative pairs that enhance model training through complementary supervision. Experiments on Flickr30K and COCO30K demonstrate that RePair achieves superior performance with significantly fewer synthetic samples compared to traditional augmentation methods, highlighting its data efficiency and effectiveness in addressing localized semantic distinctions.
Transforming retrieval failures into hard pairs can boost model performance while using 26% to 75% fewer synthetic samples than conventional methods.
Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26\%--75\% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.