Search papers, labs, and topics across Lattice.
This study extends the concept of contextual entrainment in large language models (LLMs) from individual tokens to entire sentences, revealing that sentences in prompts can significantly influence token probabilities during inference. By analyzing 26 LLMs across various tasks, the authors demonstrate that larger models exhibit reduced sentence-level contextual entrainment, which is primarily regulated by a small fraction of attention heads. The findings suggest that disabling these specific attention heads can effectively reduce contextual entrainment without compromising overall model performance.
Sentence-level contextual entrainment can skew inference probabilities, but selectively disabling just a few attention heads can mitigate this effect without sacrificing performance.
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.
Tsinghua AI