Search papers, labs, and topics across Lattice.
This paper introduces the Contrastive Denoising Autoencoder (CDAE), which enhances the robustness of pre-trained BERT embeddings against semantic-preserving perturbations like synonym substitution and word dropout. By jointly optimizing contrastive and reconstruction objectives, CDAE achieves significant improvements in embedding stability while maintaining semantic integrity. Experimental evaluations demonstrate that CDAE outperforms both original BERT embeddings and SimCSE in preserving similarity under various perturbation strategies, underscoring the effectiveness of perturbation-invariant representation learning.
CDAE boosts BERT's robustness, achieving superior embedding stability against semantic perturbations while preserving meaning.
Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
