Search papers, labs, and topics across Lattice.
This paper introduces CoSTALA, a novel training paradigm that enhances audio-language models by shifting from coarse-grained to fine-grained spatio-temporal reasoning. By employing a multi-granularity hierarchical loss function, CoSTALA effectively models temporal dependencies and accurately anchors individual acoustic events, thereby preserving their semantic integrity. Experimental results show that this approach significantly improves the alignment between auditory and textual representations in complex multi-event audio sequences.
CoSTALA achieves unprecedented accuracy in aligning multi-event audio sequences with text, overcoming the limitations of traditional coarse-grained methods.
Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.
