Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
We introduce SpecMRL, a novel redesign of Matryoshka Representation Learning (MRL) that addresses the limitation of MRL's unconstrained prefix optimization by incorporating a parameter-derived spectral basis organized by feature importance. This spectral reorganization enables more effective prefix readout, leading to superior representation compression. Experiments on MTEB demonstrate that SpecMRL consistently outperforms MRL across all compression levels, achieving parity with full-dimensional MRL at half the embedding size.
Compressing LLM embeddings by 50% without sacrificing performance is now achievable by reorganizing the feature space with a learned spectral basis.
Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.