Search papers, labs, and topics across Lattice.
This paper introduces EmpaAva, the first open-source, agentic 3D-avatar empathetic chatbot that enhances empathetic response generation (ERG) through live, face-to-face interactions. Utilizing a Tri-Agent Architecture, EmpaAva integrates perception, empathetic response planning, and embodied rendering to create a seamless user experience, achieving superior performance in emotion understanding and response quality compared to existing text-only and 2D avatar systems. The system's innovative multimodal planning allows for synchronized emotional speech and facial expressions, marking a significant advancement in empathetic AI interactions.
EmpaAva outperforms traditional chatbots by delivering real-time, emotionally aware interactions through a photorealistic 3D avatar.
This paper presents EmpaAva, to our knowledge the first open-source, agentic 3D-avatar empathetic chatbot, which carries empathetic response generation (ERG) from text-only exchanges into live, face-to-face interaction. Through a video-call-like interface, a user speaks to a 3D digital human that reads their affect from speech and optional vision, and replies with emotional speech, lip-synced facial motion, and photorealistic 3D Gaussian rendering. At its core, an LLM coordinates a Tri-Agent Architecture, in which perception, empathetic response planning, and embodied rendering form a closed loop, paired with a Response Planning layer that compiles each reply into an executable multimodal plan, keeping voice, expression, and rendering on one empathetic intent. Building on strong open-source modules, EmpaAva supplies the intelligence that binds them into one controllable, inspectable experience. In automatic and human evaluations, EmpaAva surpasses text-only, 2D talking-face, and multimodal avatar baselines in emotion understanding, response quality, and audio-visual consistency. We open-source EmpaAva with an online live demo.
