Search papers, labs, and topics across Lattice.
The paper introduces MIC, a framework to improve multi-scale representation learning by optimizing the geometry of embeddings through isotropic subspace alignment. MIC uses Soft Collapse Regularization (SCR) to reduce redundancy between prefix and residual subspaces and Spectral Isotropy Regularization (SIR) to promote hyper-spherical uniformity in low-dimensional prefixes. Experiments show MIC outperforms baselines, especially in high-compression scenarios, by generating semantically dense representations with high discriminative power.
Compressing embeddings doesn't have to mean losing information: this new method maintains discriminative power even in high-compression scenarios.
Although multi-scales representation learning enables elastic-dimension embeddings, nested subspaces often suffer from dimensional redundancy and spectral collapse. To address this, we introduce MIC, a framework that optimizes the geometric landscape of multi-granular embeddings through isotropic subspace alignment. MIC employs Soft Collapse Regularization (SCR) to mitigate redundancy between prefix and residual subspaces via cross-correlation penalties, alongside Spectral Isotropy Regularization (SIR) to ensure hyper-spherical uniformity in low-dimensional prefixes. By unifying these strategies through a self-distillation objective, MIC generates semantically dense representations that maintain high discriminative power. Our experiments demonstrate that MIC significantly outperforms standard baselines, particularly in high-compression scenarios where maintaining informational capacity is most critical.
