Search papers, labs, and topics across Lattice.
This paper introduces a data-centric multimodal alignment method to enhance fine-grained food image understanding by addressing the limitations of CLIP-style models trained on heterogeneous web-collected image-text pairs. The approach involves target-aware data selection to curate visually relevant training subsets and VLM-based caption refinement to generate accurate, visually grounded descriptions. Experimental results demonstrate that this method significantly improves retrieval performance, achieving over twice the score of traditional VLM-based retrieval while maintaining efficiency, with a notable average performance gain of approximately 19% from caption refinement alone.
Target-aware data selection and VLM-based caption refinement can boost food image retrieval performance by over 19%, revealing the potential of fine-tuning multimodal models with curated data.
Fine-grained food visual--semantic understanding requires models to capture subtle distinctions across ingredients, cooking methods, doneness, color, texture, and plate composition. Although CLIP-style vision-language models provide a natural framework for this task, their effectiveness is limited when training relies on heterogeneous web-collected image--text pairs. Such data often exhibit a web-to-target domain gap and cross-modal misalignment, where images differ from the target distribution and captions are noisy, multilingual, or weakly grounded in visual content. We propose a data-centric multimodal alignment method for fine-grained food description and recognition. Our method first performs target-aware data selection to identify visually relevant training subsets, then applies VLM-based caption refinement to generate visually grounded, target-style descriptions. Using these curated image--caption pairs, we train complementary CLIP-style retrieval experts and further combine their decisions through a hierarchical VLM-assisted multi-expert decision-level fusion strategy that invokes the VLM only when experts disagree. Experiments show that our data refinement strategy significantly improves retrieval performance over naive web supervision, with VLM-based caption refinement alone yielding an average performance gain of approximately 19%. Our full method also achieves more than twice the retrieval score of pure VLM-based retrieval while remaining substantially more efficient.
