Search papers, labs, and topics across Lattice.
This paper introduces Resonant Context Anchoring (RCA), a lightweight intervention method that enhances the factual consistency of Large Language Models (LLMs) by addressing the issue of contextual disregard during inference. RCA operates by decoupling attention routing from signal gain, allowing for the selective amplification of relevant context tokens without altering the attention probability distribution. Experimental results on the Llama-3 model series show that RCA significantly reduces factual hallucinations while preserving fluency and general language understanding, achieving a Pareto improvement in performance metrics.
RCA boosts contextual faithfulness in LLMs by amplifying relevant signals without increasing computational costs or altering attention distributions.
Large Language Models (LLMs) frequently exhibit "contextual disregard" when faced with input evidence that conflicts with their internal parametric memory, leading to persistent factual hallucinations. Existing mitigation strategies primarily rely on suppressing specific neuron activations or employing computationally expensive contrastive decoding mechanisms, which often result in increased perplexity or significantly elevated inference latency. To address these limitations, we propose Resonant Context Anchoring (RCA), a lightweight inference-time intervention method grounded in the perspective of residual stream signal dynamics. RCA aims to resolve the signal attenuation of external evidence during its propagation through deep networks. The core mechanism involves the orthogonal decoupling of routing logic and information magnitude within the self-attention module. By utilizing raw pre-softmax attention scores as an instantaneous metric of semantic alignment, we construct a dynamic gain field via non-linear rectification to selectively amplify the norms of value vectors corresponding to context tokens, without altering the attention probability distribution. This mechanism effectively elevates the signal-to-noise ratio (SNR) of input evidence within the residual stream mixture, thereby robustly anchoring the generation trajectory to the truthful context during inference. Extensive experiments on the Llama-3 model series demonstrate that RCA significantly improves contextual faithfulness across multiple factual consistency and strong knowledge-conflict tasks, effectively suppressing parametric hallucinations. Furthermore, results confirm that as a training-free and computationally negligible plug-and-play module, RCA achieves a Pareto improvement in faithfulness and fluency while maintaining the model's general language understanding capabilities.
Tsinghua AI