Search papers, labs, and topics across Lattice.
This study investigates the impact of LoRA fine-tuning on attention patterns in LLMs, specifically focusing on how task-specific relevance behavior is learned within the network. Through ablation and attention experiments, the authors demonstrate that restricting LoRA updates to a mid-network region can recover over half of the performance improvements typically gained from applying LoRA across all layers. The findings reveal a strong correlation between enhanced ranking performance and attention to key relevance-oriented features, such as rarity sensitivity and document-query interaction, thus providing insights into optimizing reranker adaptations.
Restricting LoRA updates to a compact mid-network region can recover over half of the performance gains, challenging the notion that full-layer updates are necessary for effective fine-tuning.
LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.
