Search papers, labs, and topics across Lattice.
This paper introduces CARD, an encoder-free audio captioning model that eliminates the need for an audio encoder during inference, thereby reducing computational costs and bottlenecks associated with fixed acoustic features. By distilling knowledge from a pretrained audio teacher and strategically routing its representations across different model components, CARD achieves significant performance improvements on benchmark datasets. The results show a CIDEr-D score increase of +12.18 on AudioCaps and +5.21 on Clotho, demonstrating that the placement of a teacher's knowledge is crucial for model performance, even without an encoder.
Routing a teacher's knowledge across model components can outperform traditional encoder-based approaches in audio captioning.
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.
