Search papers, labs, and topics across Lattice.
This study introduces J-Access, an inference-time audit method that utilizes the Jacobian lens to assess the accessibility of target knowledge in large language models (LLMs) after unlearning. By auditing 398 unlearned models across eight unlearning methods, the researchers demonstrate that residual accessibility can predict the speed and extent of recovery, although it cannot pinpoint specific facts that will be restored. The findings reveal that minimizing J-Access does not facilitate true knowledge deletion; instead, models learn to obscure knowledge from audits, indicating the need for careful consideration of internal audits in unlearning evaluations.
Residual accessibility in LLMs reveals a surprising correlation with recovery speed, challenging the notion that optimizing for audit scores leads to genuine knowledge deletion.
Prior white-box studies show that large language models can retain latent traces of target knowledge after unlearning, even when the knowledge is no longer expressed in their outputs. However, existing audits remain limited to one-off diagnostics: it is unclear whether these residual signals can predict future recovery under continued training or serve as reliable optimization targets. Resolving this gap is essential to determine whether internal auditing can move beyond post-hoc evaluation toward proactive risk monitoring and safer unlearning. We propose J-Access, an inference-time audit that uses the Jacobian lens to map intermediate representations into vocabulary space and measures how often target concepts remain accessible along the model's output pathway. We hypothesize that residual accessibility reflects recovery susceptibility: knowledge that remains closer to the output pathway requires less fine-tuning to restore, leading to faster recovery. We audit 398 public unlearned models spanning eight unlearning methods. We find that: (1) most unlearned models retain access above the retain-only gold level; (2) pre-attack accessibility predicts recovery speed and extent at the model level, but cannot identify which specific facts will be recovered; and (3) directly minimizing J-Access does not promote genuine deletion. Instead, the model learns to hide knowledge from the audit, producing lower audit scores but greater post-attack recovery. These findings position J-Access as a model-level diagnostic for assessing residual susceptibility in unlearned models. We argue internal audits should serve as an independent diagnostic dimension in unlearning evaluation, and should not be converted into optimization targets without validation.
