Search papers, labs, and topics across Lattice.
This paper introduces Context-driven Missing-Modality Learning (CMML), a framework for robust medical diagnosis using image and tabular data when some modalities are missing. CMML uses a Cascade Residual Transformer-based Autoencoder (CRTA) with context tokens to synthesize missing modalities and then aligns the synthesized and available representations using instance-adaptive semantic references. Experiments on three medical datasets show that CMML outperforms state-of-the-art methods, achieving average AUC improvements of over 1% across the board.
Even when crucial medical data is missing, a new framework can still achieve state-of-the-art diagnostic accuracy by intelligently synthesizing the missing modalities and aligning them with available data.
While multimodal data integrating diverse imaging and clinical tabular records is crucial for accurate medical diagnosis, the arbitrary absence of specific modalities is prevalent in clinical practice, severely degrading the performance of multimodal models. Existing methods either discard missing modalities, leading to information loss, or struggle to synthesize them without capturing complex inter-modal dependencies. To address these limitations, we propose a novel Context-driven Missing-Modality Learning (CMML) framework, which sequentially performs modality synthesis and semantic alignment to achieve robust diagnosis under arbitrary missing conditions. Specifically, we design a Cascade Residual Transformer-based Autoencoder (CRTA) that leverages learnable context tokens acting as dataset-level semantic prior to capture inter-modal dependencies and synthesize key missing representations. These representations are further enriched by modality-specific memory banks. To resolve the discrepancy between original available and synthesized representations, we transform the learned context tokens into instance-adaptive semantic references by infusing multimodal representations from the CRTA's outputs. This reference guides the alignment of heterogeneous modality representations into a unified space, where class-aware contrastive refinement is finally applied to explore discriminative diagnostic cues. Extensive evaluations on skin lesion (Derm7pt), ocular disease (ODIR), and meningioma (MEN) datasets demonstrate that CMML significantly outperforms state-of-the-art (SOTA) methods, yielding AVG AUC improvements of 1.26%, 0.97%, and 1.32%, respectively.
