Search papers, labs, and topics across Lattice.
This paper introduces Attention Head Reweighting (AHR), a novel method for adapting large language models (LLMs) to new text-classification tasks using minimal data. By learning a single scalar per attention head, AHR significantly reduces the number of trainable parameters while leveraging the functional specialization of attention heads. Experimental results demonstrate that AHR outperforms standard baselines like LoRA in data-scarce scenarios, achieving superior performance with 200-1000x fewer parameters modified.
AHR achieves state-of-the-art performance in text classification with a fraction of the parameters, revolutionizing how LLMs can adapt to limited data.
Learning effectively from limited data is critical in domains like security where labeled examples are scarce. Large language models (LLMs) have demonstrated some capabilities for data-efficient learning, especially through parameter-efficient adaptation methods, but continue to struggle when faced with few samples for difficult tasks. To meet this challenge, we propose Attention Head Reweighting (AHR), a data-efficient method that adapts LLMs to new text-classification tasks by learning only a single scalar per attention head. This drastically reduces the number of parameters that need to be learned by making use of the functional specialization of individual attention heads. Experiments on diverse open-source text classification datasets show that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as our AHR only modifies ~0.0001% of the model's parameters. In addition, our learned weights are easy to interpret and can be analyzed to better understand the mechanisms and attention heads responsible for in-context learning abilities in LLMs.
Microsoft Research