Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces QR-Erase, a novel framework for machine unlearning that employs Pivoted QR decomposition to efficiently identify and eliminate task-specific representations from model parameters. By implementing Layer-Localized QR-Erase, the method focuses updates on layers with the highest concentration of targeted information, achieving a superior forgetting-retention tradeoff compared to traditional optimization-based approaches. The results indicate that QR-Erase not only maintains performance within 5% of singular value decomposition (SVD) methods but also significantly enhances unlearning efficiency, particularly in speech tasks.
Efficiently removing task-specific information from models can be achieved with QR-Erase, which outperforms traditional methods while preserving model capabilities.
Machine unlearning seeks to remove targeted information from trained models without requiring costly retraining. Existing optimization-based methods often degrade unrelated capabilities, while subspace-based approaches rely on computationally expensive singular value decompositions (SVD). We introduce QR-Erase, a subspace-based framework that uses Pivoted QR decomposition to identify and remove task-specific representations directly from model parameters. We further propose Layer-Localized QR-Erase, which restricts updates to layers containing the highest concentration of task-specific information. We show that Pivoted QR provides accurate subspace recovery with bounded error, and that under a mild spectral gap condition, the recovered subspace approaches the optimal SVD solution. Across task-level, cross-lingual, and speech unlearning, QR-Erase achieves a stronger forgetting-retention tradeoff than optimization-based methods while remaining within 5% of SVD across all metrics. Exploiting low-rank and layer-localized structure further improves forgetting (for example, reducing speech forget-set accuracy from 53.1% to 15.7%). These results demonstrate that accurate subspace recovery, rather than optimal reconstruction, is sufficient for effective unlearning and provides an efficient and general alternative to SVD-based methods for modern foundation models.