Search papers, labs, and topics across Lattice.
This paper introduces SnapBench, a novel benchmark designed to evaluate the robustness of snap-and-ask multimodal retrieval systems under various input corruptions. By assessing 16 multimodal retrievers across 1,145 queries and 9,085 gallery items, the study reveals that image corruptions significantly hinder retrieval performance, while text corruptions primarily affect text-only retrieval with minimal impact on joint retrieval. The findings indicate that clean image-only retrieval often surpasses joint retrieval, underscoring the necessity for effective modality calibration in the presence of noisy inputs.
Image corruptions can drastically impair retrieval performance, revealing that clean image-only queries often outperform joint retrieval in snap-and-ask scenarios.
Mobile AI acts as a visual oracle, empowering users to snap a picture of something and ask for information. Snap-and-ask retrieval is now one of the most common entry points for mobile AI, yet photos are often blurry, while text questions may be short or mistyped. Existing benchmarks only test on clean inputs or do not isolate paired robustness in snap-and-ask retrieval. Therefore, we introduce SnapBench, the first paired benchmark for robust snap-and-ask multimodal retrieval, spanning 1,145 queries, 9,085 gallery items under 53 controlled corruption conditions with human annotations. We evaluate 16 multimodal retrievers, covering dual-tower encoders and embedding-based VLMs. Results show that image corruptions substantially degrade retrieval, while text corruptions mainly affect text-only retrieval and have limited impact on joint retrieval. Clean image-only retrieval often outperforms joint retrieval, indicating the coarse-text drag and the lack of cross-modal fallback under noisy inputs. SnapBench provides a controlled testbed for evaluating robust retrieval in snap-and-ask scenarios. We further propose MOOR (Modality-anchored, Outlier-aware, Optimal Reweighting), a simple adaptive fusion approach, highlighting the need for reliability-aware modality calibration in snap-and-ask retrieval.
Tsinghua AI