Search papers, labs, and topics across Lattice.
The paper introduces AMT-X, a novel phase-structured multi-turn red-teaming framework designed to evaluate the safety of large language models (LLMs) against adaptive adversaries. By employing a multi-role jury and checklist-gated evaluation, AMT-X significantly enhances the granularity of attack success measurement, distinguishing between partially actionable outputs and those with complete operational detail. The framework achieves an impressive attack success rate of 97.6-100% under lenient conditions, but reveals a concerning gap of up to 33 percentage points when stricter criteria are applied, highlighting the inadequacy of traditional evaluation methods.
AMT-X reveals that existing safety evaluations can miss up to 33% of actionable harm by conflating partial success with full operational capability.
Safety evaluation of large language models (LLMs) relies largely on single-turn attack datasets and single-judge scoring, underestimating risk from adaptive multi-turn adversaries and reporting a single success rate that does not separate partially actionable outputs from those carrying complete operational detail. We propose AMT-X (Adaptive Multi-Turn Exploitation), a phase-structured multi-turn red-teaming framework. Unlike prior multi-turn attacks that rely on ad hoc escalation or free-form per-goal plans, AMT-X casts the attack as an explicit, reproducible multi-phase state machine driven by semantic signals from the victim, and replaces single-judge scoring with a multi-role jury whose phase-conditioned checklists gate success on actionable harm. Across six frontier victim models (queried under their default safety alignment, without added moderation layers) and seven Moderation sub-categories, AMT-X attains overall attack success rates of 97.6-100% under a lenient score threshold, but 66.7-78.6% under a stricter gate requiring complete, real, and operational detail: a gap of up to 33 percentage points between partially and fully actionable harm.
