Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a multimodal framework that leverages MLLMs to improve single-image nutrition estimation by addressing silent failures in food item identification and portion estimation. The approach verifies food identities and the suitability of proposed regions for portion estimation, then uses these insights for a targeted recovery pass to fill identified gaps and omissions. This method achieves improved mass and energy accuracy without task-specific fine-tuning, enhancing item-level grounding and overall nutrition estimation performance.
MLLMs can recover missed food items and correct inaccurate portion estimations in single images, boosting nutrition estimation accuracy without task-specific fine-tuning.
Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.