Search papers, labs, and topics across Lattice.
This study introduces a real-time model for generating listener nodding in avatar dialogue systems by predicting both the timing and kinematic parameters of nodding movements. Utilizing a dyadic attention network and Voice Activity Projection (VAP), the model achieves context-aware predictions that enhance the naturalness of avatar interactions. Experimental results indicate that this approach significantly outperforms traditional methods that rely on stochastic timing or fixed-motion nodding, demonstrating its effectiveness in creating more engaging conversational avatars.
Real-time listener nodding predictions can transform avatar interactions from mechanical to genuinely engaging, enhancing user experience in dialogue systems.
In human dialogue, we achieve smooth communication by expressing nonverbal cues such as eye contact, nodding, and facial expressions with precise timing. It is expected for conversational avatars to express these cues appropriately to realize natural and human-like interactions. This study focuses on nodding, which is crucial for demonstrating active listening and encouraging further user utterances. We propose a model that predicts both timing and kinematic parameters representing the motion features of listener nodding in real time. The proposed model consists of a timing prediction module and a kinematic parameter prediction module. Each implements a dyadic attention network over the speaker and listener channels based on the technique of Voice Activity Projection (VAP). Unlike conventional models, this approach enables real-time prediction of kinematic parameters based on the specific context of the dialogue rather than just predicting the timing. Furthermore, we demonstrate the effectiveness of fine-tuning the kinematic parameter prediction module initialized from the trained timing prediction module. The proposed model is lightweight and capable of real-time operation, and it has been integrated into an avatar dialogue system. Subjective evaluation experiments shows that our proposed method significantly outperforms both a baseline with stochastic timing and another with fixed-motion nodding. The code and trained models are available at https://github.com/MaAI-Kyoto/MaAI.
