Search papers, labs, and topics across Lattice.
The paper introduces Alignment Adapter (AlAd), a lightweight, sliding-window-based adapter module designed to align token-level embeddings of compressed deep learning models with those of their larger counterparts. This alignment aims to improve the performance of compressed models by preserving local contextual semantics and enabling flexible alignment across different architectures and dimensionalities. Experiments on BERT-family models across token-level NLP tasks demonstrate that AlAd significantly enhances the performance of compressed models with minimal overhead.
Compressed models can punch *way* above their weight with Alignment Adapters, recovering significant performance on token-level tasks with minimal overhead.
Compressed Deep Learning (DL) models are essential for deployment in resource-constrained environments. But their performance often lags behind their large-scale counterparts. To bridge this gap, we propose Alignment Adapter (AlAd): a lightweight, sliding-window-based adapter. It aligns the token-level embeddings of a compressed model with those of the original large model. AlAd preserves local contextual semantics, enables flexible alignment across differing dimensionalities or architectures, and is entirely agnostic to the underlying compression method. AlAd can be deployed in two ways: as a plug-and-play module over a frozen compressed model, or by jointly fine-tuning AlAd with the compressed model for further performance gains. Through experiments on BERT-family models across three token-level NLP tasks, we demonstrate that AlAd significantly boosts the performance of compressed models with only marginal overhead in size and latency.
