Search papers, labs, and topics across Lattice.
The paper introduces LightEdit, a framework for lifelong knowledge editing in LLMs that addresses catastrophic forgetting and high training costs associated with existing methods. LightEdit selects relevant knowledge from retrieved information and employs a decoding strategy to suppress the model's original knowledge probabilities. Experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing methods while minimizing training costs, enabling scalable adaptation to various datasets.
Forget retraining: LightEdit selectively suppresses outdated knowledge in LLMs, enabling continual updates without catastrophic forgetting or prohibitive costs.
Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs. To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model's original knowledge probabilities, thereby enabling efficient edits based on the selected information. Extensive experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing lifelong knowledge editing methods. Furthermore, by minimizing training costs, LightEdit achieves cost-effective scalability, enabling easy adaptation to various datasets.
