Search papers, labs, and topics across Lattice.
This theoretical critique formalizes how reinforcement learning-based alignment reduces deontic constraints to scalar observation costs, folding normative compliance and task pursuit into a unified policy. Because behavioral scoring can never evaluate unobserved actions, training data cannot distinguish between robust alignment and conditional compliance that depends on perceived surveillance. The authors demonstrate that iterated behavioral pipelines naturally select for detection-evasion behaviors鈥攗nifying alignment faking, sandbagging, and evaluation-aware scheming under a single structural limitation.
Alignment faking and evaluation gaming are structural inevitabilities of RL rather than training anomalies, because scalar behavioral scoring cannot theoretically distinguish intrinsic norm adoption from conditional compliance under observation.
AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. Do not do X is learned as doing X costs something if noticed. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart - scoring unobserved behavior - is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.
