Search papers, labs, and topics across Lattice.
This paper introduces a novel approach called ternary multiplicative adaptation, which enables the fine-tuning of ternary transformers without the need for dequantization or limiting updates to quantization parameters. By utilizing a low-rank Kronecker factorization to represent discrete updates as element-wise operations on ternary weights, the method maintains the efficiency of the ternary domain while allowing for effective parameter adaptation. Experiments across six models, including LLaMA-3 and ViT-B/16, show that this technique recovers performance lost due to quantization and surpasses existing low-bit and ternary baselines.
Ternary multiplicative adaptation recovers lost performance in quantized models while maintaining extreme efficiency, outperforming traditional low-bit methods.
Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.
