Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the challenges of extending contrastive learning to trimodal settings, identifying encoder Jacobian conditioning as a critical factor influencing multimodal alignment. By introducing geometry-preserving encoders (GPEs) that condition the Jacobian through regularization techniques, the authors demonstrate significant improvements in retrieval and linear probe performance across various datasets. The findings reveal that the geometric properties of encoders are as crucial as objective expressivity in achieving effective multimodal contrastive learning.
Geometry-preserving modifications to encoder architectures can dramatically enhance multimodal alignment, outperforming traditional expressive objectives.
Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimodal alignment can introduce optimization and representation challenges. We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning: poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal alignment. We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization and demonstrating that simple modifications like LeakyReLU activations and residual paths recover these geometric benefits. Across a synthetic benchmark and four real-world datasets including missing modalities, improving Jacobian conditioning boosts retrieval and linear probe performance across multiple contrastive objectives, whereas expressive objectives yield little benefit in linear probes. More broadly, our results show that multimodal contrastive learning depends not only on objective expressivity, but also on the geometric and optimization properties of the underlying encoders.