Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for Automatic Audio Captioning (AAC) that integrates auxiliary AudioSet semantics to enhance the generation of natural language descriptions for complex acoustic scenes. By employing a ConvNeXt encoder to extract frame-level acoustic representations and augmenting them with top-$K$ predicted AudioSet keywords, the authors create a structured contextual basis for a compact six-layer BART-style decoder. The resulting architecture achieves competitive caption quality while significantly improving computational efficiency, making it more suitable for practical deployment compared to traditional LLM-based approaches.
Grounding audio captioning in structured AudioSet semantics allows for high-quality descriptions without the computational burden of large language models.
Automatic Audio Captioning (AAC) seeks to generate natural language descriptions of complex acoustic scenes, bridging auditory perception and language understanding. However, word-selection indeterminacy and increasing reliance on large-scale sequence-to-sequence or LLM-based models limit practical deployment. We propose a resource-efficient AAC framework that explicitly grounds caption generation in auxiliary AudioSet semantics. Frame-level acoustic representations extracted using a ConvNeXt encoder are augmented with top-$K$ predicted AudioSet keywords, providing structured contextual cues for decoding. A compact six-layer BART-style decoder conditions on this joint acoustic-semantic representation, enabling caption generation without LLM-scale decoding. The proposed design balances semantic grounding and computational efficiency within a compact architecture. Evaluations on Clotho V2 and AudioCaps confirm competitive caption quality under practical deployment constraints.
