Search papers, labs, and topics across Lattice.
This paper introduces Spatial-Omni, a novel method that integrates First-Order Ambisonics (FOA) spatial audio into multimodal large language models (LLMs) through a lightweight SO-Encoder, enhancing their ability to process spatial cues without altering existing audio encoders. The authors constructed a comprehensive dataset, SO-Dataset, along with evaluation benchmarks, enabling the assessment of 16 spatial audio understanding subtasks. Experimental results indicate that Spatial-Omni significantly outperforms current Large Audio-Language Models (LALMs) and Omni LLMs in tasks related to spatial audio understanding while maintaining general audio comprehension.
Spatial-Omni achieves superior spatial audio understanding in multimodal LLMs by effectively incorporating spatial cues without modifying existing architectures.
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at https://github.com/dieKarotte/Spatial-Omni.
