Search papers, labs, and topics across Lattice.
This paper introduces HeadRank, a decoding-free passage reranking framework that optimizes preference alignment in LLM attention heads to overcome the issue of attention score homogenization. By employing entropy-regularized head selection and a distribution regularizer, HeadRank enhances the discriminability of middle-context documents, achieving significant improvements in ranking performance. The method demonstrates a 43-percentage-point selectivity gap in identifying relevant documents, significantly outperforming existing generative and decoding-free baselines across multiple benchmarks.
HeadRank achieves a remarkable 43-percentage-point improvement in selecting relevant documents from the middle-context zone, revolutionizing passage reranking efficiency.
Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to $\mathcal{O}(1)$ forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank consistently outperforms generative and decoding-free baselines with 100\% formatting success. At 4B, 57.4\% of relevant middle-zone documents reach the top quartile versus 14.2\% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking.
