Search papers, labs, and topics across Lattice.
This paper introduces VQ-Transplant, a framework that facilitates the integration of new Vector Quantization (VQ) modules into pre-trained visual tokenizers without the need for extensive retraining. By preserving the encoder-decoder parameters during the transplantation process and employing a lightweight decoder adaptation strategy, the authors achieve near state-of-the-art reconstruction fidelity while significantly reducing training costs by 95%. This approach not only enhances the efficiency of VQ-based models but also democratizes access to advanced quantization techniques for researchers with limited resources.
VQ-Transplant slashes training costs by 95% while maintaining near state-of-the-art reconstruction quality, making advanced VQ techniques accessible to resource-constrained researchers.
Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose {\bf VQ-Transplant}, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95\%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.
