Search papers, labs, and topics across Lattice.
The paper introduces ModalImmune, a training framework designed to improve the robustness of multimodal models against modality loss or corruption. ModalImmune achieves this by selectively collapsing modality information during training, forcing the model to learn robust joint representations. The framework incorporates a spectrum-adaptive collapse regularizer, an information-gain guided controller, curvature-aware gradient masking, and a Neumann-truncated hyper-gradient procedure for meta-parameter adaptation, demonstrating improved resilience and stability on multimodal benchmarks.
Make your multimodal models immune to missing modalities with a training framework that selectively collapses modality information, boosting robustness without sacrificing performance.
Multimodal systems are vulnerable to partial or complete loss of input channels at deployment, which undermines reliability in real-world settings. This paper presents ModalImmune, a training framework that enforces modality immunity by intentionally and controllably collapsing selected modality information during training so the model learns joint representations that are robust to destructive modality influence. The framework combines a spectrum-adaptive collapse regularizer, an information-gain guided controller for targeted interventions, curvature-aware gradient masking to stabilize destructive updates, and a certified Neumann-truncated hyper-gradient procedure for automatic meta-parameter adaptation. Empirical evaluation on standard multimodal benchmarks demonstrates that ModalImmune improves resilience to modality removal and corruption while retaining convergence stability and reconstruction capacity.
