Search papers, labs, and topics across Lattice.
This paper introduces GiVA, a gradient-informed initialization strategy for vector-based adaptation, addressing the high rank requirements of existing methods. GiVA initializes adaptation vectors using gradient information, enabling comparable training times to LoRA while maintaining parameter efficiency. Experiments across NLU, NLG, and image classification show GiVA achieves competitive or superior performance to LoRA and other vector-based methods, with an 8x reduction in rank.
Vector-based fine-tuning just got an 8x speed boost, rivaling LoRA's performance with a fraction of the parameters, thanks to a clever gradient-informed initialization.
As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).
