Search papers, labs, and topics across Lattice.
This paper introduces a novel benchmark for evaluating knowledge editing techniques in large language models (LLMs), focusing on the logical consequences of fact edits rather than just the recall of edited information. By extracting logical rules from a knowledge graph and generating multi-hop questions, the study assesses how well existing methods, such as ROME and FT, can propagate knowledge beyond direct assertions. The results reveal a significant performance gap, with up to 24% lower accuracy in handling entailed knowledge, underscoring the necessity for more semantics-aware evaluation frameworks in this domain.
Existing knowledge editing methods can accurately insert facts into LLMs but often fail to capture the broader logical implications, revealing a critical oversight in current evaluation practices.
Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new benchmark designed to evaluate how knowledge editing methods handle the logical consequences of a single fact edit. Our benchmark extracts relevant logical rules from a knowledge graph for a given edit. Then, it generates multi-hop questions based on these rules to assess the impact on logical consequences. Our findings indicate that while existing knowledge editing approaches can accurately insert direct assertions into LLMs, they frequently fail to inject entailed knowledge. Specifically, experiments with popular methods like ROME and FT reveal a substantial performance gap, up to 24%, between evaluations on directly edited knowledge and on entailed knowledge. This highlights the critical need for semantics-aware evaluation frameworks in knowledge editing.
