Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper establishes a novel reconstructability metric for agent-safety evaluations, addressing the critical issue that identical nominal outcomes may arise from fundamentally different evidence bases. By implementing a counterfactual-replay intervention protocol and introducing Evidence Sufficiency Cards, the authors demonstrate that safety-evaluation claims are often unsupported by sufficient evidence, as shown by low sufficiency spans across various inputs. The findings reveal that current evaluation methods may misrepresent safety, emphasizing the need for a robust framework to validate claims through reconstructability.
Safety-evaluation claims often lack the evidence needed to support them, with a striking 0.458-0.833 sufficiency span across evaluated inputs revealing a significant gap in validation.
Many agent-safety evaluation results are not yet load-bearing evidence: identical nominal outcomes (task success, attack success, monitor scores) may sit atop materially different evidence regimes. No vendor-neutral, runnable instrument scores reconstructability as an evaluation-validity metric: whether captured evidence can reconstruct the decision a claim depends on. This paper introduces a property-level reconstructability metric over eight decision-property classes and a cross-harness adapter emitting per-decision Evidence Sufficiency Cards backing a per-run monitor-coverage release check. It specifies a counterfactual-replay intervention protocol, implements its replayability-precondition probe, and defines a claim-evidence overclaim gap. On public and bundled traces, without new model runs, twelve-field sufficiency spans 0.458-0.833 across four inputs sharing a surface reading; replay preconditions are unmet in every scored trace. In a synthetic release-gate pair, the sufficiency gate blocks the raw variant (0.542) and passes the instrumented (0.667). Safety-evaluation claims should travel with their reconstructability vector; a reproducibility package regenerates every reported number.