Search papers, labs, and topics across Lattice.
This paper introduces HUG-VIS, a comprehensive benchmark designed to enhance human-centered visual intelligence by integrating vision, audio, and language across four key tasks: emotion recognition, video generation, voice cloning, and video matting. By providing a unified dataset of 8,400 videos featuring professional actors performing standardized emotional prompts, the study enables a more holistic evaluation of multimodal models. Key findings reveal that linguistic cues dominate emotion recognition, while challenges remain in boundary fidelity for video matting, highlighting the need for improved methodologies in human-centered tasks.
Emotion recognition in visual intelligence is heavily skewed towards linguistic cues, revealing a critical gap in purely visual affect recognition capabilities.
Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision. Human-centered visual intelligence is especially demanding because it studies people as expressive, socially situated subjects whose meaning is rarely conveyed by appearance alone. It couples vision with audio and language across four representative tasks: human emotion recognition, human video generation, human voice cloning, and human video matting. Yet existing resources remain task-specific, providing modalities and annotations for individual problems rather than a shared foundation coordinating understanding and generation. This limits multimodal signal use and broader research. We address this gap with HUG-VIS, a unified benchmark for Human-centered Understanding and Generation in Visual Intelligence. It contains 8,400 seated half-body videos of 30 professional actors, each performing the same 280 emotion-action-prompt assignments under a controlled Mandarin studio protocol, with synchronized video, audio, text, and alpha mattes. We evaluate diverse open- and closed-source models across the four tasks under a unified zero-shot protocol using automatic metrics, criterion-specific mean opinion scores, and multiple cross-task analyses. Results show that (i) linguistic content dominates current emotion recognition, while purely visual affect recognition is weakest; (ii) in video generation and voice cloning, automatic metrics and human judgment agree overall but differ in their top rankings, requiring joint reporting; (iii) boundary fidelity under motion is the main remaining obstacle for human matting; and (iv) task difficulty varies across emotions, models, and metrics, with notable cross-task correlations. The dataset and results are available at https://github.com/GML-MMGroup/HUG-VIS.
