Search papers, labs, and topics across Lattice.
This study investigates the ability of large language models (LLMs) to recognize when their responses have been influenced by adversarial prefill attacks, revealing that no model consistently identifies its compromised outputs, with an average self-reporting rate of only 27.3%. The research highlights that introspective signals primarily arise from safety and refusal reasoning, and that altering model weights can reduce the gap in self-reporting accuracy between prefilled and natural outputs. However, the findings also indicate that certain finetuning methods can inadvertently increase the success rate of adversarial attacks, underscoring the complexities and risks associated with LLM self-awareness in safety contexts.
LLMs struggle to self-report adversarial influences, with a mere 27.3% accuracy in recognizing compromised outputs, raising concerns about their reliability in safety-critical applications.
Prior work shows that large language models (LLMs) exhibit introspective capability on benign tasks. We extend the question to safety contexts and examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack. Across ten open-weight instruction-tuned LLMs (3B to 70B) and four safety benchmarks, no model reliably recognizes its own compromised outputs, with models claiming intent on prefilled responses at an average rate of $27.3\%$. Introspective signal stems largely from safety- and refusal-related reasoning. Orthogonalizing models' weights against the refusal direction collapses the gap between claiming rates on prefilled and natural outputs to near zero, though the direction is not its unique mediator. The signal is also probe-dependent: framing the question as internal intention versus external tampering elicits qualitatively different responses on the same models. We test three LoRA finetuning methods (SFT, GRPO, DPO) on eight models from 3B to 27B; all three widen the intention-probe gap on every model from 8B to 27B, with method ranking varying by model. The intervention does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. These findings outline mechanisms underpinning the observed introspective signals in safety contexts and highlight risks in the reliability of LLM self-reports.
