Search papers, labs, and topics across Lattice.
This paper introduces CogniRoute, a schema-guided Mixture-of-Experts framework designed to enhance social video question answering by effectively routing social evidence across multiple modalities. By leveraging a cognitive schema that organizes examples based on cross-modal relations and temporal scope, CogniRoute optimizes both token generation and expert allocation through route-aware reinforcement learning. The model demonstrates significant improvements in accuracy, achieving 59.38% on the newly constructed OmniSocialBench, particularly excelling in tasks that require audio-visual coordination and temporal reasoning.
CogniRoute outperforms existing models by over 15 percentage points in social video QA, revealing the critical role of cognitive schema in multimodal reasoning.
Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.
