Search papers, labs, and topics across Lattice.
The paper introduces ID-LoRA, a parameter-efficient fine-tuning (PEFT) method that addresses the trade-off between parameter efficiency and performance in LoRA when adapting large language models (LLMs) to complex multi-task settings. ID-LoRA extracts and reuses clustered parameter groups from the pre-trained weight matrix to form multiple low-rank components, sharing a single trainable low-rank matrix. Experiments across five benchmarks demonstrate that ID-LoRA outperforms full fine-tuning and existing PEFT baselines, achieving better performance with significantly fewer trainable parameters, particularly in multi-task scenarios.
ID-LoRA slashes trainable parameters by up to 46% compared to standard LoRA while boosting performance across diverse benchmarks, offering a sweet spot between efficiency and effectiveness for fine-tuning LLMs.
LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.
