Search papers, labs, and topics across Lattice.
This paper introduces MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework designed to tackle incomplete multimodal learning (IML) by leveraging the deviations in representations caused by missing modalities. By training a residual router to guide expert specialization based on the differences between task representations from incomplete and complete inputs, the method effectively captures and utilizes the privileged residual signal. Experimental results demonstrate that MARS outperforms existing baselines in multimodal classification and segmentation tasks while maintaining efficiency and adaptability across various architectures.
Missing modalities can enhance model performance by guiding expert specialization, revealing a new avenue for improving incomplete multimodal learning.
As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train-test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router's decisions when the feature router deviates, enhancing expert robustness under imperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines while remaining efficient and extensible to diverse backbones and tasks.
