Search papers, labs, and topics across Lattice.
The paper investigates how fine-tuning LLMs on new knowledge impacts their ability to retrieve previously stored knowledge, leading to hallucinations. They fine-tune Llama 3.1 8B, Gemma 2 9B, and Mistral 7B v03 on single QA datasets, controlling the amount of new knowledge and training epochs, and observe increased hallucinations with more new knowledge and longer training. They then introduce Monotonic Relationship Feature Identification (MoRFI), a method to identify causally relevant latent directions in the residual stream using sparse autoencoders, revealing that exposure to new facts disrupts the retrieval of stored knowledge along specific directions.
LLMs' factual recall falters when fine-tuned on new information, and this can be traced to specific latent directions in the residual stream.
Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction. Subsequent stages of post-training often introduce new facts outwith the parametric knowledge, giving rise to hallucinations. While it has been demonstrated that supervised fine-tuning (SFT) on new knowledge may exacerbate the problem, the underlying mechanisms are still poorly understood. We conduct a controlled fine-tuning experiment, focusing on closed-book QA, and find latent directions that causally contribute to hallucinations. Specifically, we fine-tune Llama 3.1 8B, Gemma 2 9B and Mistral 7B v03 on seven distinct single QA datasets, controlling for the percentage of new knowledge and number of training epochs. By measuring performance on the test set, we validate that incrementally introducing new knowledge increases hallucinations, with the effect being more pronounced with prolonged training. We leverage pre-trained sparse autoencoders (SAEs) to analyze residual stream activations across various checkpoints for each model and propose Monotonic Relationship Feature Identification (MoRFI) for capturing causally relevant latents. MoRFI filters SAE features that respond monotonically to controlled fine-tuning data mixtures of a target property. Our findings show that exposure to unknown facts disrupts the model's ability to retrieve stored knowledge along a set of directions in the residual stream. Our pipeline reliably discovers them across distinct models, recovering knowledge through single-latent interventions.
