Search papers, labs, and topics across Lattice.
This paper introduces RoleJudge, an evaluation framework using audio large language models to assess character alignment in speech dialogue systems by analyzing both textual responses and vocal features. They create RoleChat, a voice role-playing dataset with chain-of-thought annotations, and use it to train RoleJudge with a multi-stage training paradigm and reinforcement learning incorporating Standard Alignment. Experiments show RoleJudge outperforms baselines in accuracy and subjective assessments, demonstrating its effectiveness in multidimensional character evaluation.
Audio LLMs can now be systematically evaluated for character alignment in role-playing scenarios, thanks to a new framework that judges both text and vocal features.
The rapid evolution of multimodal large models has revolutionized the simulation of diverse characters in speech dialogue systems, enabling a novel interactive paradigm. Character attributes are manifested not only in textual responses but also through vocal features, as speech conveys rich paralinguistic information that is challenging to quantify. This poses significant difficulties in evaluating the character alignment of role-playing agents. To address these challenges, we present RoleJudge, an evaluation framework that leverages audio large language models to systematically assess the alignment between speech and character across multiple modalities and dimensions. Furthermore, we introduce RoleChat, the first voice role-playing evaluation dataset enriched with chain-of-thought reasoning annotations, comprising a diverse set of authentic and LLM-generated speech samples. Utilizing this dataset, we implement a multi-stage training paradigm and incorporate Standard Alignment in reinforcement learning to mitigate reward misalignment during optimization. Experimental results in terms of accuracy and subjective assessment demonstrate that RoleJudge outperforms various baseline models, validating the effectiveness of our multidimensional evaluation framework.
