Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Adaptive Hierarchical Representation Alliance (AHRA), a novel framework that addresses the semantic granularity mismatch in multimodal models by factorizing inputs into shared and private streams across different semantic levels. By employing layer-wise CKA analysis, the authors reveal that traditional alignment methods overlook the varying depths at which different modalities provide task-relevant information, leading to reduced reliability in challenging conditions. Experimental results across six benchmarks demonstrate that AHRA significantly outperforms strong baselines, particularly in scenarios with noisy or missing inputs.
AHRA reveals that aligning multimodal inputs at the same semantic depth can lead to significant performance drops, especially under noisy conditions.
Multimodal models often align language, vision, and audio in a single final-layer latent space, implicitly assuming that task-relevant evidence emerges at the same semantic depth across modalities. Using layer-wise CKA analysis, we observe that this assumption leads to semantic granularity mismatch: textual cues usually require deeper contextual abstraction, whereas visual and acoustic cues often provide discriminative perceptual evidence in shallow or middle layers. This mismatch can flatten fine-grained modality-private cues and reduce reliability under noisy, imbalanced, or missing inputs. To address this, we proposed Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework. AHRA factorizes each modality into shared and private streams across semantic levels, regularizes them with shared alignment and private decorrelation, routes shared information through a cross-modal expert, and enhances task-relevant private tokens with modality-specific experts guided by a sparsity-controlled soft-gating mechanism (foreground exam). A hierarchical co-fusion module then performs intra-level expert coordination and inter-level semantic selection. Experiments on six benchmarks across image-text classification, multimodal intent recognition, and trimodal sentiment analysis show that AHRA consistently improves over strong baselines and remains robust under noisy and missing-modality settings.