Search papers, labs, and topics across Lattice.
SceneBind introduces an omni-modal representation that integrates semantic and 3D spatial understanding across vision, audio, and language, addressing the limitations of existing models that focus primarily on instance-level semantics. By representing scenes as semantic-spatial entities with global embeddings and object-centric slots, SceneBind captures both object-level semantics and spatial attributes, enhancing the model's ability to understand complex scenes. The proposed SceneBind Matching scheme enables effective cross-modal scene retrieval and object grounding, achieving state-of-the-art performance while facilitating zero-shot transfer to tasks like audio-visual localization.
SceneBind revolutionizes scene understanding by seamlessly integrating what and where across multiple modalities, achieving state-of-the-art retrieval and grounding capabilities.
We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.
