Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Logit-Contribution Scoring (LOCOS), a novel method for identifying attention heads in large language models that engage in non-literal retrieval during long-context tasks. By focusing on the output-value (OV) circuit rather than literal token matching, LOCOS effectively distinguishes heads that synthesize answers from relevant context spans. The results show that ablating the top LOCOS heads significantly degrades performance on the NoLiMa benchmark, indicating that these heads are crucial for non-literal retrieval, with a marked drop in ROUGE-L scores compared to traditional methods.
Ablating just 50 retrieval-specific heads identified by LOCOS can reduce model performance from a ROUGE-L score of 0.401 to virtually zero, highlighting their critical role in non-literal answer synthesis.
In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.