Search papers, labs, and topics across Lattice.
This paper introduces Orthogonal Representation Editing (ORE), a novel approach to batch knowledge editing in Large Language Models (LLMs) that addresses the challenges posed by semantic representation entanglement. By enforcing orthogonal constraints on edit vectors within a constructed semantic subspace, ORE significantly enhances the precision of knowledge updates while minimizing performance degradation. Experimental results demonstrate that ORE not only outperforms existing methods but also excels in cross-lingual knowledge editing tasks, showcasing its robustness and versatility.
Orthogonal Representation Editing achieves unprecedented precision in knowledge updates by decoupling semantic entanglement, outperforming traditional methods in both efficiency and accuracy.
Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.
