Search papers, labs, and topics across Lattice.
This paper introduces ActorMindBench, a new benchmark for speech role-playing, and ActorMind, a multi-agent reasoning framework designed to emulate human actor reasoning. ActorMind uses separate agents to process role descriptions, comprehend emotional cues in spoken dialogue, generate emotional states, and deliver scripts infused with those states. Experiments demonstrate that ActorMind effectively enhances speech role-playing capabilities.
Unlock more realistic human-machine interaction with ActorMind, a framework that enables AI to convincingly role-play via speech by reasoning about roles, scenes, and emotional cues like a human actor.
Role-playing has garnered rising attention as it provides a strong foundation for human-machine interaction and facilitates sociological research. However, current work is confined to textual modalities, neglecting speech, which plays a predominant role in daily life, thus limiting genuine role-playing. To bridge this gap, we conceptualize and benchmark speech role-playing through ActorMindBench, and we present a corresponding reasoning framework, called ActorMind. Specifically, (1) Speech Role-Playing enables models to deliver spontaneous responses with personalized verbal traits based on their role, the scene, and spoken dialogue. (2) ActorMindBench is a hierarchical benchmark comprises Utterance-Level content with 7,653 utterances, Scene-Level content with 313 scenes, and Role-Level content with 6 roles. (3) ActorMind is an off-the-shelf, multi-agent, chain-of-though style reasoning framework that emulates how human actors perform in theaters. Concretely, ActorMind first reads its assigned role description via Eye Agent, then comprehends emotional cues within contextual spoken dialogues through Ear Agent. Subsequently, Brain Agent generates a descriptive emotional state, and finally, Mouth Agent delivers the scripts infused with corresponding emotion state. Experimental results demonstrate the effectiveness of ActorMind in enhancing speech role-playing.
