Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Higher-order Representation and Information Learning (HRIL) to explicitly model synergistic information in multimodal representations by leveraging higher-order statistical dependencies. HRIL constructs an empirical cross-moment tensor and uses Tucker decomposition with a synergy-aware regularizer to preserve the information capacity for signals emerging only from joint modality configurations. Experiments show HRIL consistently outperforms existing contrastive methods, particularly on tasks heavily reliant on synergistic interactions.
Synergistic multimodal information, previously elusive, is unlocked by modeling higher-order statistical dependencies via tensor decomposition.
Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.