Search papers, labs, and topics across Lattice.
This paper introduces ParaPairAudioBench, a novel benchmark consisting of 5,175 audio pairs that evaluates Large Audio-Language Models (LALMs) on five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. The study reveals that current LALM judges underperform relative to human judgments, with an average deficit of 32 percentage points and significant calibration issues, particularly in cases where abstention is the correct choice. By incorporating both same-transcript and cross-transcript conditions, this benchmark facilitates a nuanced analysis of LALM performance in paralinguistic evaluations, highlighting areas for improvement in model reliability.
LALMs are falling short in paralinguistic evaluations, missing the mark by 32 percentage points compared to human judges.
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
