Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a human-machine collaborative framework for Vietnamese Speech Emotion Recognition (SER) that leverages LLM-based reasoning to address challenges stemming from ambiguous acoustic patterns and limited annotated data. Acoustic feature-based models provide auxiliary signals (confidence, feature-level evidence) to guide LLMs, and a confidence-based routing mechanism delegates uncertain samples to LLMs for deeper reasoning based on human-derived rules. Experiments on a Vietnamese speech dataset demonstrate that this approach achieves up to 86.59% accuracy, outperforming purely data-driven methods, particularly in handling ambiguous cases.
Human-guided reasoning with LLMs can boost Vietnamese speech emotion recognition accuracy by intelligently routing ambiguous samples for deeper, rule-based analysis.
Vietnamese Speech Emotion Recognition (SER) remains challenging due to ambiguous acoustic patterns and the lack of reliable annotated data, especially in real-world conditions where emotional boundaries are not clearly separable. To address this problem, this paper proposes a human-machine collaborative framework that integrates human knowledge into the learning process rather than relying solely on data-driven models. The proposed framework is centered around LLM-based reasoning, where acoustic feature-based models are used to provide auxiliary signals such as confidence and feature-level evidence. A confidence-based routing mechanism is introduced to distinguish between easy and ambiguous samples, allowing uncertain cases to be delegated to LLMs for deeper reasoning guided by structured rules derived from human annotation behavior. In addition, an iterative refinement strategy is employed to continuously improve system performance through error analysis and rule updates. Experiments are conducted on a Vietnamese speech dataset of 2,764 samples across three emotion classes (calm, angry, panic), with high inter-annotator agreement (Fleiss Kappa = 0.8574), ensuring reliable ground truth. The proposed method achieves strong performance, reaching up to 86.59% accuracy and Macro F1 around 0.85-0.86, demonstrating its effectiveness in handling ambiguous and hard-to-classify cases. Overall, this work highlights the importance of combining data-driven models with human reasoning, providing a robust and model-agnostic approach for speech emotion recognition in low-resource settings.