Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Contrastive Learning (HCL), a novel framework for multimodal representation learning that moves beyond the typical shared-private decomposition by modeling globally shared, partially shared, and modality-specific representations. HCL uses a hierarchical latent-variable model with structural sparsity and a structure-aware contrastive objective to align modalities based on shared latent factors, with theoretical guarantees for identifiability and recovery under uncorrelated latent variables. Experiments on synthetic and real-world multimodal electronic health records demonstrate HCL's ability to recover hierarchical structure and improve predictive performance compared to existing methods.
Stop forcing all your modalities into a shared-private representation: HCL learns which modalities *actually* share information, leading to better representations and predictions.
Multimodal representation learning is commonly built on a shared-private decomposition, treating latent information as either common to all modalities or specific to one. This binary view is often inadequate: many factors are shared by only subsets of modalities, and ignoring such partial sharing can over-align unrelated signals and obscure complementary information. We propose Hierarchical Contrastive Learning (HCL), a framework that learns globally shared, partially shared, and modality-specific representations within a unified model. HCL combines a hierarchical latent-variable formulation with structural sparsity and a structure-aware contrastive objective that aligns only modalities that genuinely share a latent factor. Under uncorrelated latent variables, we prove identifiability of the hierarchical decomposition, establish recovery guarantees for the loading matrices, and derive parameter estimation and excess-risk bounds for downstream prediction. Simulations show accurate recovery of hierarchical structure and effective selection of task-relevant components. On multimodal electronic health records, HCL yields more informative representations and consistently improves predictive performance.
