Search papers, labs, and topics across Lattice.
This study investigates the limitations of Activation Oracles (AOs), which are designed to interpret internal activations of language models. Through a Taboo Word Guessing experiment, the authors reveal that fine-tuned AOs can develop concept-specific blind spots, failing to retrieve concepts that are otherwise decodable within their internal representations. This finding highlights a critical reliability issue in using AOs for model interpretability, as their outputs may not accurately reflect the underlying information they are trained to access.
Fine-tuned Activation Oracles can become concept-specific anti-readers, failing to retrieve information that is otherwise accessible within their own internal states.
Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.
