Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of current reasoning methods in audio-visual social understanding by analyzing the IntentBench benchmark and introducing IntentBench-Prime, which addresses the high noise levels in the original dataset. The authors find that a straightforward Vanilla SFT baseline not only matches but often outperforms more complex reasoning approaches across multiple benchmarks, highlighting the inefficiency of existing methods. Additionally, they demonstrate that substantial social understanding can be derived from textual data alone, challenging the reliance on video input for performance.
A simple Vanilla SFT model outperforms complex reasoning methods in social audio-visual question answering, revealing the inefficiencies of current approaches.
Training Multimodal Large Language Models for audio-visual social understanding is a crucial step toward embodied social intelligence. Chain-of-thought (CoT) reasoning has become the dominant approach, with HumanOmniV2 and its IntentBench benchmark as a prominent reference point. In this context, we report three findings. First, IntentBench is highly noisy: $\sim$7% of questions are broken and $\sim$23% are trivially answerable without the video input. We remove the affected questions and release Intentbench-Prime. Second, current reasoning approaches are expensive and surprisingly ineffective. A simple Vanilla SFT baseline matches or outperforms existing reasoning methods across three benchmarks at a fraction of the cost, establishing it as an essential baseline for evaluating novel fine-tuning techniques. Third, our analysis reveals that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT. These surprising findings reveal the limitations of current MLLMs when it comes to social understanding. IntentBench-Prime, Vanilla SFT model, and code are publicly available.
