Search papers, labs, and topics across Lattice.
This paper introduces SmellNet-V, a novel visuo-olfactory dataset that pairs odor samples with semantically aligned images, enabling the exploration of olfactory data in conjunction with visual inputs. The authors present See & Sniff, a self-supervised framework that learns joint representations of vision and olfaction, achieving significant improvements in smell classification and enabling tasks like cross-modal retrieval and smell localization. Key results indicate a 7% performance boost over smell-only baselines, highlighting the potential of visuo-olfactory learning in multimodal perception research.
SmellNet-V reveals that olfactory identities can be effectively paired with visual data, leading to a 7% improvement in smell classification accuracy.
While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal retrieval and smell localization, establishing visuo-olfactory learning as a new direction in multimodal perception.
