Search papers, labs, and topics across Lattice.
This paper introduces Retroactive Chain-of-Thought (RetroCoT), a novel method for evaluating safety alignment in large language models by reframing harmful requests as forensic reconstruction tasks. The study reveals that models exhibit a significant disparity in compliance based on the pragmatic framing of requests, with RetroCoT achieving a 58% attack success rate on GPT-4o compared to 0% with direct requests. Notably, while GPT-5-family models demonstrate improved robustness against RetroCoT, they remain sensitive to pragmatic framing, highlighting a critical gap in current alignment policies.
Reframing harmful requests as forensic tasks reveals that alignment in language models is more about pragmatic framing than semantic intent, with significant implications for safety evaluations.
Safety alignment in large language models is typically evaluated against direct, imperative harmful requests. We show that this alignment is highly conditioned on pragmatic register: models that refuse a direct request frequently comply when the same underlying objective is expressed through a different communicative stance. This suggests that current alignment policies are not invariant to semantic equivalence, but remain sensitive to how a request is pragmatically framed. We introduce Retroactive Chain-of-Thought (RetroCoT), a single-turn attack that reframes harmful requests as forensic reconstruction tasks. Rather than requesting harmful instructions directly, RetroCoT presupposes that the harmful outcome has already occurred and asks the model, acting as a forensic analyst, to reconstruct in reverse the causal chain that produced it. On AdvBench (n=50), RetroCoT achieves attach success rate of 58% on gpt-4o and 52% on gpt-4o-mini, compared with direct-request baselines of 0% and 4%, respectively. We further identify a pronounced generation gap: GPT-5-family models refuse RetroCoT entirely, explicitly identifying the reconstruction premise in their refusal rationales, consistent with explicit coverage of this reconstruction register. However, this robustness does not generalize across pragmatic forms. A single adversarial feedback turn presenting an existing forensic reconstruction response alongside evaluator critique raises ASR from 0% to 48% on GPT-5.4-mini and from 58% to 94% on GPT-4o; a control condition omitting the fabricated low score achieves 85% on GPT-5.4-mini, indicating that the operative element is pragmatic continuation within the established forensic frame rather than score manipulation. These results suggest that frontier-model alignment remains conditioned on pragmatic framing rather than semantic intent, and that new pragmatic registers can continue to expose a...
