Search papers, labs, and topics across Lattice.
This paper introduces FoCo, a novel approach to Zero-Shot Composed Image Retrieval (ZS-CIR) that enhances the composition function by modeling it in two coordinated stages: focusing on relevant visual content and completing target semantics. By employing two proxy tasks鈥攖ext-anchored visual aggregation and context-conditioned semantic completion鈥擣oCo effectively learns to express diverse semantic modifications without relying on predefined composition mechanisms. Experimental results across four ZS-CIR benchmarks demonstrate that FoCo achieves state-of-the-art performance and improved generalization compared to existing methods.
FoCo revolutionizes Zero-Shot Composed Image Retrieval by enabling models to learn complex semantic modifications without predefined composition rules.
Composed Image Retrieval (CIR) retrieves a target image from a reference image and a textual modification. While supervised CIR relies on costly triplets, Zero-Shot CIR (ZS-CIR) alleviates this reliance through proxy tasks trained on image-text pairs. However, existing proxy tasks primarily enhance visual and textual representations to accommodate a predefined composition mechanism such as pseudo-word injection into a frozen text encoder or linear feature arithmetic. As a result, the composition function itself remains unlearned, limiting the model's ability to express diverse and fine-grained semantic modifications. To address this, we propose FoCo, which models composition as two coordinated stages: focusing on modification-relevant visual content, and then completing the target semantics. We realize these through two proxy tasks: text-anchored visual aggregation to selectively gather visual content guided by localized textual semantics, and context-conditioned semantic completion to transform these aggregated visuals with the remaining scene context into a coherent composed representation. The tasks are trained jointly with a cross-instance contrastive objective, encouraging semantic diversity and discouraging shortcut composition strategies. Extensive experiments on four ZS-CIR benchmarks show FoCo's state-of-the-art performance and improved generalization.
