Search papers, labs, and topics across Lattice.
This paper introduces a novel approach called Learning from Reliable Latent Prompts, which addresses the challenge of performance degradation in large-scale multimodal models (LMMs) when faced with missing modality inputs. By hypothesizing that learnable latent prompts can serve as stable, modality-intrinsic priors decoupled from corrupted inputs, the authors demonstrate that these prompts can effectively guide cross-modal knowledge compensation even under extreme conditions of missing data. Empirical evaluations across three benchmark datasets reveal that this method achieves state-of-the-art performance, significantly outperforming existing prompt learning strategies in scenarios with high missing rates.
Learnable latent prompts can stabilize multimodal models and maintain performance even when up to 90% of input modalities are missing.
Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities. In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation. Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features, regardless of whether the input modalities are complete or partially absent. However, such input-conditioned strategies are hindered by the escalating unreliability of instance-level features; as higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance. To address this limitation, we hypothesize that learnable latent prompts themselves encapsulate stable, modality-intrinsic priors that are decoupled from corrupted inputs. Consequently, we propose a novel paradigm: Learning from Reliable Latent Prompts. Unlike prior methods, we model input-agnostic learnable prompts as stable latent anchors that enable robust guidance and effective cross-modal knowledge compensation, even under extreme missing rates (e.g., 90%). Empirical results across three benchmark datasets demonstrate that our "learn-from-latent-prompts" approach achieves state-of-the-art performance across a wide range of missing-modality scenarios. Extensive experiments further confirm the effectiveness of this paradigm in providing a robust solution to the missing-modality problem.
