Search papers, labs, and topics across Lattice.
This paper introduces an efficient adaptation of the Recursive Feature Machine (RFM) algorithm to identify multi-dimensional refusal subspaces in Large Language Models (LLMs), addressing the computational challenges of existing methods. By utilizing a probe-informed initialization, the authors demonstrate that their approach can extract these subspaces in seconds, significantly improving performance on both reasoning and non-reasoning models. The findings suggest that RFM not only accelerates subspace identification but also outperforms traditional methods in ablation tasks, indicating its potential as a scalable tool for enhancing safety and interpretability in LLMs.
Identifying multi-dimensional refusal subspaces in LLMs can be done in seconds, unlocking new avenues for safety and interpretability without the computational burden.
Steering and monitoring activations in Large Language Models (LLMs) are increasingly used for both safety and interpretability. Early work assumed behaviours are encoded along single linear directions, but recent findings suggest complex behaviours, such as the refusal to answer harmful queries, live in multi-dimensional subspaces. However, existing methods for extracting these subspaces are computationally expensive, which becomes prohibitive on reasoning models who produce long reasoning traces. By adapting the Recursive Feature Machine (RFM) algorithm -- which can be computed efficiently -- with a probe-informed initialization, we are able to identify the multi-dimensional refusal subspace in seconds, on reasoning (Qwen 3) and non-reasoning (Qwen 2.5) models. While RFM allows for faster subspace identification, it also showed better performances on the ablation task than its alternatives. More work is planned to better understand the relations between subspaces found by different methods. If confirmed, RFM could be a cheap and scalable complement to existing subspace-extraction methods in LLMs.
