Search papers, labs, and topics across Lattice.
This paper introduces SAGA, a novel framework that leverages frozen multimodal large language models (MLLMs) to enhance visual embedding training by providing attribute-aware supervision. By employing Group Relative Policy Optimization (GRPO), SAGA encourages the vision encoder to focus on specific attributes that distinguish or unify image pairs, moving beyond traditional scalar-based training methods. The approach significantly improves zero-shot image retrieval performance, achieving a Recall@1 increase of 3 to 6 points over state-of-the-art methods across multiple datasets.
Semantic attribute gradients from frozen MLLMs can boost zero-shot image retrieval performance by up to 6 points, reshaping how visual embeddings are trained.
Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched. A multimodal large language model (MLLM), shown the same pair, can articulate those attributes and use them to predict whether the images share a class. We propose SAGA, a framework that turns this language-grounded, attribute-aware perception into a training signal for the encoder itself. Specifically, we use Group Relative Policy Optimization (GRPO) to reward the MLLM for correct predictions on the vision encoder's tokens. Since correct predictions require those tokens to expose the specific attributes that differ or match between the pair, the gradient pushes the encoder to encode them, replacing the uniform pair-level scalar with attribute-resolved supervision. An auxiliary attention-distillation loss anchors the encoder's embedding to tokens the MLLM attended to, and a standard metric-learning loss shapes the embedding geometry for nearest-neighbour retrieval. The MLLM is frozen throughout and discarded at inference, matching the deployment cost of a metric-learning baseline. SAGA improves Recall@1 by 3 to 6 points over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves on zero-shot image retrieval.
