Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces CoSTALA, a novel training paradigm that enhances audio-language models by shifting from coarse-grained to fine-grained spatio-temporal reasoning. By employing a multi-granularity hierarchical loss function, the model effectively captures temporal dependencies and anchors individual acoustic events, thereby preserving their semantic integrity. Experimental results show that CoSTALA outperforms existing methods in processing multi-event audio sequences, marking a significant advancement in audio-language alignment capabilities.
Fine-grained spatio-temporal reasoning in audio-language models can dramatically enhance the understanding of complex multi-event audio sequences.
Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.