Search papers, labs, and topics across Lattice.
This paper introduces FreezeEmpath, an innovative approach for training empathetic spoken chatbots that utilizes frozen large language models (LLMs) and existing speech instruction data. By avoiding finetuning and catastrophic forgetting, FreezeEmpath effectively leverages speech emotion recognition (SER) data to enhance emotional expressiveness in generated responses. Experiments show that this method not only improves empathetic dialogue performance but also surpasses existing models in SER and SpokenQA tasks, highlighting its efficiency and effectiveness.
FreezeEmpath achieves superior empathetic dialogue capabilities without the need for costly finetuning, relying instead on frozen LLMs and existing data.
Empathy is essential for fostering natural interactions in spoken dialogue systems, as it enables machines to recognize the emotional tone of human speech and deliver empathetic responses. Recent research has made significant progress in developing empathetic spoken chatbots based on large language models (LLMs). However, several challenges still exist when training such models, including reliance on costly empathetic speech instruction data and a lack of emotional expressiveness in the generated speech. Finetuning LLM with cross-modal empathetic instruction data may also lead to catastrophic forgetting and a degradation of its general capability. To address these challenges, we propose FreezeEmpath, an end-to-end empathetic spoken chatbot trained in a simple and efficient manner. The entire training process relies solely on existing speech instruction data and speech emotion recognition (SER) data, while keeping the LLM's parameters frozen. Experiments demonstrate that FreezeEmpath is able to generate emotionally expressive speech and outperforms other empathetic models in empathetic dialogue, SER, and SpokenQA tasks, demonstrating the effectiveness of our training strategy.
