Search papers, labs, and topics across Lattice.
This paper introduces two multimodal modules, LIM-F and LIM-Q, that enhance food image segmentation by integrating ingredient labels derived from large language models (LLMs) into the visual analysis pipeline. By allowing direct injection of semantic information without the need for pre-alignment of images and text, the method significantly improves performance on the FoodSeg103 benchmark, achieving a mean Intersection over Union (mIoU) of 55.0 with the Mask2Former decoder and Swin-L encoder. The approach not only outperforms existing models on visually similar and rare food categories but also maintains a modest increase in GPU memory usage, making it a scalable solution for fine-grained food understanding.
Injecting LLM-derived ingredient labels into food image segmentation boosts accuracy to state-of-the-art levels while keeping resource demands low.
Food image segmentation plays a vital role in health-related applications such as nutrition tracking and personalized health monitoring. However, existing models often underperform on visually similar ingredients and rare food categories. To address this issue, we propose two plug-and-play multimodal modules that enhance the segmentation performance by leveraging ingredient labels inferred from food images using large language models (LLMs). The first module, called LIM-F (Language Injection Module for Features), is designed to pair with any image encoder that produces multi-layer outputs (e.g., Swin Transformer), while the second module, LIM-Q (Language Injection Module for Queries), targets Mask2Former-style Transformer-based decoders. Both modules enable training without the need for pre-aligning images with text by directly injecting semantic ingredient information into the visual analysis pipeline. On the FoodSeg103 benchmark, the proposed method achieves state-of-the-art performance. Specifically, integrating LIM-Q into the Mask2Former decoder with a Swin-L image encoder yields a mean Intersection over Union (mIoU) of 55.0. LIM-F also demonstrates strong generalization and competitive performance, reaching an mIoU of 54.4 under the same model (Swin-L+Mask2Former). Furthermore, its applicability extends beyond Transformer-based decoders, as evidenced by an improvement from 47.7 to 49.8 mIoU when integrated into a CNN-based architecture. Notably, the improved segmentation accuracy is achieved with only a moderate (at most 3.8 GB) increase in the GPU memory consumption during training. Thus, the proposed approach offers a practical and scalable solution for fine-grained food understanding.
