Search papers, labs, and topics across Lattice.
This study investigates whether activation probes from open-weight AI coding models can reveal security vulnerabilities in code generated by closed-weight models, which are otherwise opaque to human reviewers. By fitting linear probes on a dataset of vulnerable and fixed Python functions, the authors demonstrate that these probes can effectively identify vulnerabilities, scoring higher on the original vulnerable code than on its fixed counterpart in 61-67% of cases. This approach outperforms traditional prompting methods, indicating that model activations contain critical security signals that are overlooked when relying solely on output logits or written verdicts.
Activation probes can uncover security vulnerabilities in AI-generated code that traditional prompting methods completely miss.
AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.
