Search papers, labs, and topics across Lattice.
This paper introduces a Mixture-of-Modality-Experts (MoME) framework with Holistic Token Learning (HTL) to improve multimodal visual analytics, particularly for driver action recognition. MoME adaptively combines modality-specific experts, while HTL enhances both intra-expert refinement and inter-expert knowledge transfer using class and spatio-temporal tokens. Experiments on a public benchmark demonstrate that MoME with HTL outperforms existing single-modal and multimodal baselines, showing improved subtle multimodal understanding and interpretability.
Adaptive expert collaboration and holistic token learning can significantly boost performance in fine-grained multimodal visual analytics tasks like driver action recognition.
Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for decision-making.Existing multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal fusion.We validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.
