Search papers, labs, and topics across Lattice.
The paper introduces IConE (Instance-Contrasted Embeddings), a self-supervised learning framework that prevents representation collapse in Joint-Embedding Architectures (JEAs) without relying on batch statistics. IConE maintains a global set of learnable auxiliary instance embeddings regularized by an explicit diversity objective, transferring the anti-collapse mechanism to a dataset-level embedding space. Experiments across 2D and 3D biomedical modalities demonstrate that IConE outperforms contrastive and non-contrastive baselines in small-batch regimes (B=1 to B=64) and exhibits robustness to class imbalance, while also preserving high intrinsic dimensionality in the learned representations.
Train robust self-supervised models even with batch size 1: IConE prevents representation collapse by using a global set of learnable instance embeddings instead of relying on batch statistics.
Self-supervised learning (SSL) has revolutionized representation learning, with Joint-Embedding Architectures (JEAs) emerging as an effective approach for capturing semantic features. Existing JEAs rely on implicit or explicit batch interaction -- via negative sampling or statistical regularization -- to prevent representation collapse. This reliance becomes problematic in regimes where batch sizes must be small, such as high-dimensional scientific data, where memory constraints and class imbalance make large, well-balanced batches infeasible. We introduce IConE (Instance-Contrasted Embeddings), a framework that decouples collapse prevention from the training batch size. Rather than enforcing diversity through batch statistics, IConE maintains a global set of learnable auxiliary instance embeddings regularized by an explicit diversity objective. This transfers the anti-collapse mechanism from the transient batch to a dataset-level embedding space, allowing stable training even when batch statistics are unreliable, down to batch size 1. Across diverse 2D and 3D biomedical modalities, IConE outperforms strong contrastive and non-contrastive baselines throughout the small-batch regime (from B=1 to B=64) and demonstrates marked robustness to severe class imbalance. Geometric analysis shows that IConE preserves high intrinsic dimensionality in the learned representations, preventing the collapse observed in existing JEAs as batch sizes shrink.
