Search papers, labs, and topics across Lattice.
The paper introduces Cross-Level Co-Representation (CLCR), a novel approach for multimodal learning that addresses semantic misalignment by explicitly modeling a three-level semantic hierarchy within each modality. CLCR uses Intra-Level Co-Exchange Domains (IntraCED) to factorize features into shared and private subspaces at each level, restricting cross-modal attention to the shared subspace, and Inter-Level Co-Aggregation Domains (InterCAD) to synchronize semantic scales and selectively fuse shared representations. Experiments across six benchmarks demonstrate that CLCR achieves state-of-the-art performance in tasks such as emotion recognition and action recognition, indicating its effectiveness and generalizability.
By explicitly modeling a multi-level semantic hierarchy and carefully controlling information exchange between modalities, CLCR achieves state-of-the-art results in multimodal learning tasks ranging from emotion recognition to action recognition.
Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.
