Search papers, labs, and topics across Lattice.
This study refines universal audio representations by integrating audio-description alignment into an acoustically pretrained encoder, enhancing its ability to convey high-level concepts across diverse audio domains. By comparing matched control, shuffled-description, and correctly paired trajectories, the authors establish that correct alignment significantly boosts classification performance, achieving a 4.66-point improvement with a linear probe and a 2.59-point increase with a sequence-aware LLM. Notably, the findings reveal that correct pairing accounts for 87% of the linear-probe gain, underscoring the importance of semantic alignment in audio representation tasks.
Correctly aligning audio descriptions with representations can boost classification performance by nearly 5 points, revealing the critical role of semantic refinement in audio tasks.
Universal audio representations must preserve acoustic detail while making high-level concepts accessible across speech, music, environmental sound, and downstream models of different capacities. We study semantic refinement of an acoustically pretrained encoder by adding audio-description alignment to a foundation of BEST-RQ, reconstruction, and CTC. We compare matched control, shuffled-description, and correctly paired trajectories to distinguish correct correspondence from an extra contrastive objective. Each endpoint is frozen and evaluated with a temporal-mean linear probe and a sequence-aware LLM readout, testing whether the refined information is directly accessible and remains useful to a stronger model. Across three paired seeds, correct alignment improves domain-balanced classification by 4.66 points with the linear probe and 2.59 points with the sequence-aware LLM, with positive changes in every domain. Correct pairing accounts for 87% of the linear-probe gain, while the LLM shows its clearest correspondence-specific benefit in captioning. Dense acoustic objectives provide complementary gains under both readouts. A separate 24-layer continuation remains competitive with leading public encoders under the shared evaluator, supporting the recipe beyond the controlled study.
