Search papers, labs, and topics across Lattice.
This paper introduces Spatial-Omni, a novel method that integrates First-Order Ambisonics (FOA) spatial audio into multimodal large language models (LLMs) through a lightweight SO-Encoder, enhancing spatial audio understanding without altering existing audio encoders. By constructing the SO-Dataset, SO-QA, and SO-Bench, which include 400K FOA spatial audio clips and 2.1M spatial question-answer pairs, the authors provide a robust framework for training and evaluating spatial audio tasks. Experimental results demonstrate that Spatial-Omni significantly outperforms current large audio-language models on various spatial understanding tasks while maintaining general audio comprehension.
Spatial-Omni achieves superior spatial audio understanding by seamlessly integrating FOA encoding into existing LLMs, outperforming traditional models without compromising general audio processing.
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at https://github.com/dieKarotte/Spatial-Omni.
