Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates how LLMs track and retrieve information from context, finding that they exhibit a serial-recall-like pattern, assigning peak probability to tokens immediately following repeated tokens. Through ablation studies, the authors demonstrate that induction heads, which attend to the token following a previous occurrence of the current token, play a critical role in this behavior. Removing high-induction-score heads significantly reduces the +1 lag bias and impairs serial recall performance, establishing a mechanistic link between these heads and temporal context processing.
LLMs exhibit a surprising "serial recall" bias, favoring tokens that immediately follow repeated tokens in the input sequence, and it's specifically driven by induction heads.
Large language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in transformers, suggesting that these heads are especially important for ordered retrieval and serial-recall-like behavior during in-context learning.