Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TLoRA+, a parameter-efficient fine-tuning method that integrates the TLoRA optimizer into LoRA, enhancing performance without significantly increasing computational cost. TLoRA+ leverages the efficiency of low-rank adaptation while improving upon it through optimizer-level modifications. Experiments on the GLUE benchmark demonstrate the effectiveness and robustness of TLoRA+ across various model architectures.
TLoRA+ boosts LoRA's performance without adding computational overhead, offering a simple path to better fine-tuning.
Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.