Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a training-free method for representation selection and dimensionality reduction in image encoders using cross-model agreement via Canonical Correlation Analysis (CCA). By finding linear projections that maximize the correlation between representations from two pre-trained encoders, the method retains shared semantic content while discarding redundant dimensions. Experiments on multiple datasets demonstrate that this approach can reduce representation dimensionality by over 75% while improving downstream task performance, achieving accuracy gains of up to 12.6% compared to baselines and PCA.
Ditch the training and still shrink your image embeddings by 75% while boosting accuracy, thanks to a clever CCA trick that distills knowledge between pre-trained models.
Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.