Search papers, labs, and topics across Lattice.
This study investigates the concept of computational provenance, focusing on whether generated text can embed detectable evidence of the internal states that influenced its production. By employing two distinct neural architectures鈥攁 modular feed-forward network and a transformer model鈥攖rained on an arithmetic task with mandatory intermediate states, the authors successfully authenticate and recover signals related to these internal states from the generated text. The findings demonstrate that both architectures can consistently preserve causal information, even when the final output remains unchanged, thus validating the potential for verifiable internal state representation in language model outputs.
Generated text can carry detectable evidence of the internal computations that produced it, revealing a new dimension of model interpretability.
A language model's output does not by itself provide verifiable evidence about the internal computation that produced it. We study computational provenance: whether generated text can carry detectable evidence of which causally relevant internal state occurred. We test a bounded form of this idea in two controlled architectures: a modular feed-forward neural network and a transformer-based model. Both architectures are trained on the same arithmetic task with a mandatory pathway through two discrete intermediate states, allowing different internal paths to produce the same answer. We deliberately switch between these paths, authenticate the state actually used, and let that verified state determine a subtle statistical pattern in the generated text that can later be detected. The feed-forward and transformer systems each passed all 128 matched pairs in both their public and separately sealed protected end-to-end evaluations, with the detector recovering the signal associated with the authenticated internal state. The required causal computation also reproduced across five independently trained feed-forward models and three independently trained transformers. In a separate answer-only transformer experiment, our linear probes did not recover a naturally learned intermediate state. These results provide a controlled proof of concept that information about a verified, causally relevant internal state can be preserved in generated text even when the answer is unchanged.
