Search papers, labs, and topics across Lattice.
This paper critiques the necessity of multimodal emotion recognition (MER) models exceeding 1B parameters, proposing a lightweight framework called Light-MER that utilizes knowledge distillation to enhance performance and efficiency. By transferring knowledge from a larger teacher model to a smaller student model, Light-MER preserves rich multimodal emotion reasoning while significantly improving deployment efficiency on resource-constrained devices. The authors introduce innovative optimization strategies, including a novel optimal transport loss and a multi-reward optimization approach, demonstrating that their method achieves state-of-the-art performance across nine benchmark datasets with reduced computational costs.
Smaller multimodal emotion models can outperform their larger counterparts while being more efficient, challenging the prevailing belief that bigger is always better.
Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc. However, these performance improvements are often accompanied by an increase in model parameter size (e.g, at least 7B), which simultaneously incurs high computational costs and reduces inference efficiency, thereby hindering real-time deployment on resource-constrained platforms such as robots and mobile devices. This raises a fundamental question: do we really need the multimodal MER model larger than 1B parameters for high-quality MER? In this paper, we challenge the assumption that larger models are inherently necessary and proposes a lightweight MER framework (called Light-MER), which achieves better and faster multimodal sentiment understanding and recognition through knowledge distillation. It can transfer knowledge from a strong, large-scale teacher model to a lightweight sub-billion-parameter student model, aiming to preserve rich multimodal emotion reasoning and recognition while substantially improving deployment efficiency. Specifically, we introduce two new optimization strategies to enhance knowledge transfer: (1) a new optimal transport loss that combines Sliced Wasserstein Distance with hidden-state alignment, and (2) a new multi-reward optimization strategy based on GRPO that balances MER performance and efficiency, aimed at further enhancing the learning capabilities of student models. Extensive experiments on nine benchmark datasets demonstrate that Light-MER achieves state-of-the-art performance while significantly improving inference efficiency. This highlights the strong potential of small multimodal emotion language models for future research. Code is available at https://github.com/GAIR-Lab/Light-MER.
