Search papers, labs, and topics across Lattice.
JASTIN is introduced as a framework for zero-shot audio and speech evaluation, addressing limitations in current objective metrics and MLLMs. It connects a frozen audio encoder with a fine-tuned LLM via a trainable adapter, using a novel data preparation pipeline (Multi-Source, Multi-Task, Multi-Calibration, Multi-Description). JASTIN achieves state-of-the-art correlations with human ratings across diverse audio tasks, surpassing general MLLMs without task-specific fine-tuning.
LLMs can now evaluate audio as well as humans, without task-specific training, thanks to a new instruction-driven framework.
The rapid advancement of generative audio models has outpaced the development of robust evaluation methodologies. Existing objective metrics and general multimodal large language models (MLLMs) often struggle with domain generalization, zero-shot capabilities, and instructional flexibility. To address these bottlenecks, we propose JASTIN, a generalizable, instruction-driven audio evaluation framework that formulates audio assessment as a self-instructed reasoning task. JASTIN bridges a frozen high-performance audio encoder with a fine-tuned LLM backbone via a trainable audio adapter. To ensure robust zero-shot generalization, we introduce a comprehensive instruction following data preparation pipeline, incorporating Multi-Source, Multi-Task, Multi-Calibration, and Multi-Description data. Experimental results demonstrate that JASTIN achieves state-of-the-art Pearson and Spearman correlations with human subjective ratings. It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks without the need for task-specific retraining.
