Search papers, labs, and topics across Lattice.
This reproducibility study critically evaluates the null-space constrained projection method, AlphaEdit, originally introduced by Fang et al. (2025) for knowledge editing in language models. While the study successfully reproduces AlphaEdit's reported metrics on LLaMA3, GPT2-XL, and GPT-J, it uncovers limitations in its generalizability to newer architectures and reveals that performance degrades significantly with increased editing counts, challenging the robustness of its theoretical guarantees. The findings highlight that AlphaEdit's effectiveness is sensitive to both model architecture and the scale of sequential edits, raising important considerations for its practical application in knowledge editing tasks.
AlphaEdit's theoretical guarantees against catastrophic forgetting are not as unconditional as claimed, revealing critical sensitivities to model architecture and editing scale.
Fang et al. (2025) introduced a null-space constrained projection, named AlphaEdit, for locate-then-edit knowledge editing methods, theoretically guaranteeing that edits do not disrupt previously preserved knowledge, and reports substantial gains over existing editing methods on LLaMA3, GPT2-XL, and GPT-J. In this work, we present a reproducibility study of AlphaEdit, reproducing its reported results under the original experimental setup and extending the evaluation along three axes: new model architectures, additional downstream benchmarks, and substantially longer sequential editing horizons. We successfully reproduce AlphaEdit's reported metrics across the original models, though we identify a discrepancy in the reported fluency and consistency metric. Extending AlphaEdit to newer model families, we find that its advantage does not generalize uniformly, which we trace to architectural assumptions in the locate-then-edit paradigm that are violated by these newer models. We further stress-test AlphaEdit's central sequential-editing claim by extending the number of edits well beyond those evaluated in the original paper, and find that performance, which is stable at the originally reported scale, degrades as edits reach a much higher count, indicating that the null-space projection's protection against catastrophic forgetting is bounded rather than unconditional. Finally, we extend evaluation of edited models on three extra benchmarks, namely, BoolQ, HellaSwag, and XSTest, and we find that large-scale sequential editing degrades both general downstream task competence and safety-relevant refusal behavior. Our results confirm that AlphaEdit performs as reported within its original scope, while showing that its core theoretical guarantees are sensitive to model architecture and editing scale in ways that have practical implications for its deployment.
