Search papers, labs, and topics across Lattice.
The paper investigates the challenge of editing rule-level knowledge in LLMs, which requires consistency across multiple interdependent forms (formulas, descriptions, instances). Through causal tracing on an extended RuleEdit benchmark, they find that rule knowledge is distributed across different transformer layers, with formulas/descriptions in earlier layers and instances in middle layers. Based on this finding, they propose Distributed Multi-Layer Editing (DMLE), which applies separate updates to early and middle layers, achieving significant improvements in rule-level editing performance compared to single-layer editing methods.
Editing a rule in an LLM is not like editing a fact; you can't just tweak one layer – formulas live in early layers, instances in the middle.
Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances. Yet most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention. This assumption does not hold for rule-level knowledge, where a single rule must remain consistent across multiple interdependent forms. We investigate this problem through a mechanistic study of rule-level knowledge editing. To support this study, we extend the RuleEdit benchmark from 80 to 200 manually verified rules spanning mathematics and physics. Fine-grained causal tracing reveals a form-specific organization of rule knowledge in transformer layers: formulas and descriptions are concentrated in earlier layers, while instances are more associated with middle layers. These results suggest that rule knowledge is not uniformly localized, and therefore cannot be reliably edited by a single-layer or contiguous-block intervention. Based on this insight, we propose Distributed Multi-Layer Editing (DMLE), which applies a shared early-layer update to formulas and descriptions and a separate middle-layer update to instances. While remaining competitive on standard editing metrics, DMLE achieves substantially stronger rule-level editing performance. On average, it improves instance portability and rule understanding by 13.91 and 50.19 percentage points, respectively, over the strongest baseline across GPT-J-6B, Qwen2.5-7B, Qwen2-7B, and LLaMA-3-8B. The code is available at https://github.com/Pepper66/DMLE.
