Search papers, labs, and topics across Lattice.
This study introduces AffectLoop, a multimodal dialogue system for empathetic social robots that integrates both speaker and listener emotional dynamics during interactions. By tracking verbal and facial affective cues, the system conditions response generation on these affective streams, resulting in a closed affective loop that enhances empathetic engagement. Evaluation against a baseline system revealed that AffectLoop significantly improved user satisfaction and emotional alignment, highlighting the importance of dynamic emotional modeling in robotic interactions.
Empathetic robots that understand and respond to emotional dynamics can significantly enhance user satisfaction and emotional alignment during interactions.
Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, existing empathetic dialogue systems are often text-centered and primarily model empathy as a one-way mapping from the user's emotion to the system response, limiting their ability to capture embodied speaker--listener affective exchange. We present AffectLoop, a multimodal speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot. The system tracks the speaker's verbal and facial affective dynamics, estimates the robot listener's own verbal and behavioral affective state, and conditions LLM-based response generation on both affective streams. The robot then generates a short spoken empathetic response together with emotionally congruent embodied behavior, forming a closed speaker--listener affective loop. We evaluate the system in a pilot within-subject study with five participants, comparing it with an otherwise identical utterance-conditioned baseline that omits the speaker- and listener-affective-state inputs. The proposed system received higher overall impression ratings, especially for empathetic response and user satisfaction. Post-hoc log analysis further showed higher speaker-listener affective alignment and stronger valence-based distress recovery. These preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.
