Search papers, labs, and topics across Lattice.
This paper introduces a novel approach called code surrogate gradient to fine-tune low-bit quantized models while maintaining deployment efficiency. By addressing the limitations of existing methods that suffer from estimation errors or inefficiencies, the authors demonstrate that their approach accelerates optimization in the quantized code space. Experimental results across various tasks, including arithmetic reasoning and instruction following, reveal that GradCodes significantly enhances the performance of low-bit models without compromising their deployment fidelity.
Fine-tuning low-bit models can be both efficient and deployment-faithful, thanks to a new optimization technique that leverages code surrogate gradients.
Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.
