Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SCENT, a multimodal framework that leverages language guidance to align olfactory representations with visual scenes, addressing the challenge of contextual environmental factors that influence smell but are not visible in images. By employing Vision-Language Models (VLMs), the authors generate scene descriptors that capture both visual and olfactory cues, enabling a smell encoder to map electronic-nose signals into a shared embedding space. Experiments on the New York Smells dataset show that SCENT significantly enhances crossmodal retrieval tasks, achieving state-of-the-art performance while providing interpretable olfactory representations that disentangle complex smell mixtures.
SCENT reveals that contextual semantic information is crucial for grounding olfactory perception, outperforming traditional vision-only approaches in smell retrieval tasks.
Images tell us what a scene looks like, but rarely what it would feel like to be there. While recent datasets pair visual scenes with electronic-nose measurements, aligning smell signals with images remains challenging because many olfactory cues arise from contextual environmental factors that are not directly visible in pixels. We introduce SCENT, a multimodal framework that uses language guidance as a semantic bridge between vision and olfaction. Our approach leverages Vision-Language Models (VLMs) to generate scene descriptors capturing objects, environmental context, and plausible ambient smell cues suggested by the visual scene. These descriptors provide semantic guidance for learning olfactory representations. We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations, and introduce a languageguided latent decomposition that separates object-specific odors from contextual environmental contributions. Experiments on the New York Smells dataset demonstrate that SCENT significantly improves crossmodal retrieval compared to vision-only baselines, achieving state-of-theart performance on smell-to-image and smell-to-text retrieval tasks. In addition, our framework produces interpretable olfactory representations that enable the disentanglement of complex smell mixtures. Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning and pave the way for future research in this area.