Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper formalizes Natural Language Autoencoders (NLAs) as referential games, distinguishing between their training (generative) and oversight (discriminative) phases. The analysis predicts that NLAs will excel at identifying which activation an explanation refers to, but struggle to discriminate true accounts from plausible false ones due to training objectives that don't enforce exclusion. Experiments confirm this, showing that while NLAs can identify their target activation, they fail to distinguish true accounts from closely matched false ones with high confidence, even with a maximally sympathetic surrogate monitor.
Explanations from Natural Language Autoencoders can identify their target activation but fail to discriminate it from plausible rivals, suggesting current training paradigms may not yield robust oversight tools.
Natural Language Autoencoders (NLAs) are proposed as oversight tools for language models: an Activation Verbaliser (AV) writes an English explanation of an activation vector, and an Activation Reconstructor (AR) maps the explanation back to a vector. Their value for oversight depends on a human monitor being able to tell a true account of an activation from a false one, given the Av's explanation. We formalise NLA training and NLA oversight deployment as two different referential games from Emergent Communication: training is a generative referential game, while oversight is a discriminative, multi-modal, partially-observable referential game played over a many-to-many relation between the monitored language model's activations and the accounts that are true of them. The gap between the two games predicts that AV's explanations will identify which activation they describe without discriminating its true account from closely matched false ones, since training never asks explanations to rule anything out. Using the NLA's own AR as a maximally-sympathetic surrogate monitor model, we experimentally confirm this theoretical insight on one released checkpoint: true accounts beat accounts of other activations in all 320 comparisons, but beat fluent rivals differing in a single claim only 60.2% of the time (std.err. 6.7%, chance at 50%, $n = 35$). A meaning-preserving rewrite by an uninvolved model costs only 2.1% of the distance to another activation's account, so the failure is not due to the NLA pair having developed private conventions in the AV's explanations. Our rival accounts test only transcript-level truths, which the monitored language model's conversation settles, so these results bound the easiest case and leave claims about the monitored model's own processing untested. We state conjectures that would make both results consequences of the framework.